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SPECIFICATION 
TITLE OF THE INVENTION 

VIRTUAL VIEWPOINT IMAGE GENERATION METHOD 
AND THREE-DIMENSIONAL IMAGE DISPLAY METHOD AND 
5 APPARATUS 

TECHNICAL FIELD 

The present invention relates to a 
technology for estimating information on a three- 
10 dimensional shape of an object from plural images to 
generate an image using the information. The 
technology of the present invention can be applied 
to a system that supports visual communications such 
as a TV phone and the like, 

15 

BACKGROUND ART 

From the past, in the fields of the 
computer graphics (CG) and the virtual reality (VR) , 
technologies have been activity studied for 

20 generating an image of an object r by a computer , 

viewed not only from a viewpoint position at which a 
camera is placed but also from a viewpoint position 
desired by a user. 

For example , there are methods for 

25 displaying a three-dimensional image of an object or 
generating an image (to be referred to as a virtual 
viewpoint image hereinafter) of an object viewed 
from a virtual viewpoint using plural images of the 
object taken under different conditions. 

30 As a method for displaying a three- 

dimensional image of an object, there is a method 
using a display having plural image display planes 
such as a DFD (Depth-Fused 3-D) display, for example. 
The DFD is a display in which plural image display 

35 planes are layered at some intervals (for example, 
refer to document 1: Japanese Patent No. 3022558). 
The DFD can be roughly classified into a brightness 
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modulation type and a transmission type. 

For displaying an image of the object on 
the DFD, a two-dimensional image of the object is 
displayed on each image display plane. At this time, 
5 if the DFD is the brightness modulation type, 

brightness of each of pixels overlapping when viewed 
from a predetermined viewpoint of an observer 
(reference viewpoint) is set in a ratio according to 
the shape of the object in the depth direction for 

10 displaying the pixels. Accordingly, as to a point 
existing on the object, brightness of the pixel on 
an image display plane existing near from the 
observer becomes large, and as to another point, the 
brightness of the pixel on a display plane existing 

15 farther from the observer becomes large. As a 

result, the observer who observes images displayed 
on each image display plane of the DFD can see a 
stereoscopic image (three-dimensional image) of the 
object. 

20 When the DFD is the transmission type, 

transmittance of each of pixels, on each image 
display plane, overlapping when viewed from a 
predetermined viewpoint of the observer (reference 
viewpoint) is set according to the shape of the 

25 object in the depth direction so as to display the 
pixels • 

In addition to the displaying method using 
the DFD, as the method for displaying the three- 
dimensional image of the object, there is a method 

30 for displaying two images having parallax 

corresponding to an interval of right and left eyes 
of the observer on a screen such as a liquid crystal 
display and 'the like. 

For generating the images for displaying 

35 the three-dimensional image of the object or for 
generating the image of the object viewed from 
arbitrary viewpoints, when the three-dimensional 
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shape of the object is known since the object is 
generated by a computer graphics and the like, for 
example, each of the images can be generated using 
the model. On the other hand, when the three- 
5 dimensional shape of the object is not known, it is 
necessary to obtain the three dimensional shape of 
the object, namely, a geometrical model of the 
object before generating each image. 

In addition, also when generating the 

10 virtual viewpoint image using the plural images, it 
is necessary to obtain the geometrical model of the 
object based on the plural images first . The 
geometrical model of the object is represented as a 
set of basic figures called polygon or voxel, for 

15 example. 

There are various methods for obtaining 
the geometrical model of the object based on the 
plural images, and many studies are being performed 
as Shape from X in the field of the computer vision* 
20 In the Shape from X, the stereo method is a 

representative model obtaining method (refer to 
document 2 : Takeo Kanade et al . : "Virtualized 
Reality: Constructing Virtual Worlds from Real 
Scenes," IEEE MultiMedia, Vol.4, No.l, pp. 34-37, 
25 1997, for example). 

In the stereo method, the geometrical 
model of the object is obtained based on plural 
images of the object taken from different viewpoints. 
At this time, a distance from the 
30 reference viewpoint for obtaining the model to each 
point of the object is calculated using 
triangulation techniques by performing corresponding 
point matching, that is, by associating points 
(pixels) on each image. But, the geometrical model 
35 of the object is not immediately obtained using the 
stereo method. A group of points on the surface of 
the object is obtained. Therefore, it is necessary 
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to determine structural information indicating how 
the points included in the point group are connected 
and what surface is formed in order to obtain the 
geometrical model of the object (refer to document 
5 3 : Katusi Ikeuchi "Model generation of real object 
using images", Journal of the Robotics Society of 
Japan, Vol.16, No. 6, pp. 763-766, 1998, for example). 

That is, in the method for obtaining the 
geometrical object using the stereo method, the 
10 apparatus (computer) for generating the image should 
perform complicated processing such as application 
of the shape of the object, statistical processing 
and the like. Therefore, high computing power is 
necessary. 

15 in addition, as the method for obtaining 

the geometrical model of the object based on the 
plural images, there is a method called Shape from 
Silhouette for determining a region that the object 
occupies in the space based on an outline of the 
20 object in each image taken from plural viewpoints 
(to be referred to as Shape from Silhouette method 
hereinafter) (refer to document 4: Potmesil, 
M: "Generating Octree Models of 3D Objects from 
their Silhouettes in a Sequence of Images," CVGIP 40, 
25 pp. 1-29, 1987, for example). 

In many cases, the geometrical model of 
the object obtained by the Shape from Silhouette 
method is represented as a set of small cubes called 
voxels. However, when the geometrical model of the 
30 object is represented by the voxels, large amount of 
data are required for representing the three- 
dimensional shape of the object. Therefore, high 
computing power is required for obtaining the 
geometrical model of the object using the Shape from 

35 Silhouette method. 

Therefore, in recent years, instead of 
representing the geometrical model of the object 
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using polygons or voxels like the stereo method and 
the Shape from Silhouette method, a method is 
proposed in which partial images of the object are 
texture-mapped to projection planes having a multi- 
layered structure so as to represent the three- 
dimensional shape of the object on the multi-layered 
planes (refer to document 5: Jonathan Shade et 
al.: "Layered Depth Images," SIGGRAPH98 Conference 
Proceedings, pp. 231-242, 1998, and document 6: Tago, 
Nitta, inamura, Harashima, -Video-Based Rendering 
using dynamic layer representation", Three- 
dimensional image conference 2001, pp. 33-36, 2001, 

for example) . 

The texture mapping is a method for 
15 setting the projection planes of the multi-layered 
structure, and mapping each partial image (texture 
image) cut out from the taken image to a projection 
plane corresponding to a distance of the object 
appearing in the texture image so as to obtain 
•20 stereoscopic visual effects. Thus, this method has 
advantages in that adequately high-speed processing 
can be performed even by graphics hardware in a 
generally widespread personal computer and in that 
handling of data is easy. 
25 However, on the other hand, when 

representing the geometrical model of the object 
using the multi-layered planes based on the texture 
mapping, if intervals at which the projection planes 
are set are too wide, detailed shape of the object 
cannot be represented. Therefore, a contrivance is 
proposed in which a value (depth value) in addition 
to color information of R (red), G (green) and B 
(blue) is added for each pixel of the texture image 
for detailed shape, for example, while rough shape 
35 is represented by the projection planes (planes). 
In the document 5, a method is proposed in which 
positions of pixels of each texture image are 
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changed according to the depth value so as to 
represent detailed depths that cannot be fully 
represented only by the multi-layered planes. In 
addition, in the document 6, a method is proposed in 
5 which transmittance of each pixel is set according 
to the depth value to represent detailed depths that 
cannot be fully represented only by the multi- 
layered planes . 

10 DISCLOSURE OF THE INVENTION 

PROBLEM TO BE SOLVED BY THE INVENTION 

However, in the above-mentioned methods 
for obtaining the geometrical model of the object, 
the method for obtaining the geometrical model of 
15 the object using the stereo method is susceptible to 
the shape of the object and the texture of the 
surface of the object, and surrounding environment 
of the object so that highly reliable information is 
not necessarily obtained for any shape of the object 
20 and for every point of the object (refer to document 
7 : Masatoshi Okutomi: M Why is stereo difficult?", 
Journal of the Robotics Society of Japan, Vol.16, 
No. 6, pp. 39-43, 1998, for example). 

When estimating the distance of the object, 
25 if the reliability of the estimation is low, there 
is a case that erroneous distance is estimated. 
When the estimated distance is erroneous, 
discontiguous noise appears in positions at which 
the estimation of the distance is erroneous in a 
30 virtual viewpoint image generated using the 

geometrical model of the object, for example. 
Therefore, the virtual viewpoint image appears that 
a lost region 7B occurs in a part of the object 1 as 
shown in Fig.l. In addition, according to the 
35 principle of the Shape from Silhouette method, the 

geometrical model of the object is obtained assuming 
that the object is gibbous. Therefore, there is a 



2005112^168 2mn ITOH INTERNATIONAL PATENT OFFICE 



NO, 7708 P. 17 



-7- 



10 



20 



25 



30 



problem in that a correct model of the object cannot 
be obtained when the object is wholly or partially 
concave . 

In addition, according to the Shape from 
Silhouette method, it is difficult in itself to 
correctly extract the outline of the object against 
the background on the image. Thus, the method for 
correctly extracting the outline is a main research 
task in the computer vision field now as in the past. 
That is, the geometrical model of the object 
obtained by the Shape from Silhouette method is a 
model obtained from an inaccurate outline, and the 
reliability is not high. Therefore, there is a 
problem in that the image generated from the 
15 geometrical model of the object obtained by the 
Shape from Silhouette method does not have 
sufficiently satisfactory quality. 

In addition, as to the method for 
representing the three-dimensional shape of the 
object using multi-layered planes like the texture 
mapping, the depth value provided for each texture 
pixel is known. That is, the method assumes that 
the shape of the object is accurately obtained. 
Therefore, when the shape of the object is not known, 
it is necessary to obtain the geometrical model of 
the object first. As a result, when there is an 
unreliable portion in the shape estimation of the 
object, there is a case that the texture image is 
mapped to an incorrect projection plane, so that 
there is a problem in that the generated image is 
remarkably deteriorated. 

in addition, in the method for 
representing the three-dimensional shape of the 
object using texture mapping, processing for mapping 
35 images on the multi-layered projection planes can be 
performed at high speed. However, higher processing 
ability is required for accurately obtaining the 
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shape of the object in the processing for obtaining 
the depth values. 

As described above, according to the 
conventional technologies, when there is an 
5 unreliable portion in estimation of the shape of the 
object, estimation of distance is tend to be 
incorrect, so that there is a problem in that 
discontiguous noise occurs on the generated image 
and the image quality is deteriorated. 

10 In addition, it is necessary to increase 

reliability of estimation for avoiding deterioration 
of image quality due to estimation error of the 
shape of the object. For this purpose, it is 
necessary to use many images and perform strict 

15 computational processing so as to obtain the 

accurate geometrical model of the object ♦ However, 
in this case, high processing ability (computational 
power) is necessary for the apparatus that generates 
the virtual viewpoint image and the like- Thus, 

20 there is a problem in that it is difficult for the 
generally used personal computer to generate the 
virtual viewpoint image having few deterioration at 

high speed. 

In addition, for increasing reliability of 
25 the geometrical model of the object, pictures taken 
from more viewpoints are necessary. Thus, there is 
a problem in that the image taking apparatus becomes 
large and the apparatus configuration is complicated. 

An object of the present invention is to 
30 provide a technology that can decrease remarkable 

deterioration of image quality that occurs at a part 
where the reliability of estimation of the shape of 
the object is low when generating the image of the 
object by obtaining the three-dimensional shape of 
35 the object from plural images. 

Another object of the present invention is 
to provide a technology that can generate the image 
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having small partial deterioration of image quality 
in a short time even by using an apparatus having 
low processing performance when generating the image 
of the object by obtaining the three-dimensional 
5 shape of the object from plural images . 

Still another object of the present 
invention is to provide a technology that can 
downsize an image-taking apparatus for taking images 
used for obtaining the geometrical model of the 

10 object so as to simplify the apparatus configuration. 

Before-mentioned and other objects, and 
new features of the present invention will become 
more apparent from the following detailed 
description when read in conjunction with the 

15 accompanying drawings, 

MEANS FOR SOLVING THE PROBLEM 

To solve the above-mentioned problem, the 
present invention is configured as a virtual 
20 viewpoint image generation method including: 

a step of obtaining plural images of an 
object taken by plural cameras; 

a step of determining a virtual viewpoint 
that is a position from which the object is viewed? 
25 and 

a step of generating a virtual viewpoint 
image that is an image of the object viewed from the 
virtual viewpoint based on the obtained images of 
the object, 

30 the step of generating the virtual 

viewpoint image including: 

a step 1 of setting projection planes 
having a multi-layered structure; 

a step 2 of obtaining each corresponding 
35 point, on the images of the object, corresponding to 
a projection point of a projection plane; 

a step 3 of determining color information 
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or brightness information of each projection point 
based on color information or brightness information 
of corresponding points; 

a step of calculating, for each of the 
5 projection points overlapping when viewed from a 
reference viewpoint in a space, a degree of 
probability that the object exists at a distance 
corresponding to a position of the projection point 
based on a degree of correlation of the 
10 corresponding points or neighborhoods of the 
corresponding points; 

a step 5 of performing mixing processing 
on color information or brightness information of 
reference points overlapping when viewed from the 
15 virtual viewpoint according to the degree of 

probability of existence of the object so as to 
determine color information or brightness 
information of each pixel of the virtual viewpoint 
image; and 

20 a step 6 of repeating steps from the step 

1 to the steps 5 for every point corresponding to 
pixels of the virtual viewpoint image. 

In addition, the image generation method 
of the present invention can be configured as an 
2 5 image generation method including: 

a step of obtaining images of an object 
taken from different viewpoints; 

a step of obtaining a three-dimensional 
shape of the object based on the images; and 
30 a step of generating an image of the 

object viewed from a viewpoint of an observer based 
on the obtained three-dimensional shape of the 
object, 

the step of obtaining the three- 
35 dimensional shape of the object including: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
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dimensional space; 

a step of determining a reference 
viewpoint for obtaining the three-dimensional shape 
of the object; 

5 a step of determining color information or 

brightness information of projection points that are 
points on the projection planes based on color 
information or brightness information of 
corresponding points, on the obtained images, 
10 corresponding to the projection points; 

a step of calculating correlation degrees 
among corresponding points corresponding to the 
projection points? 

a step of calculating, for each of the 
15 projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that the object exists at the 
projection point based on the correlation degree of 
each projection point; 
20 the step of calculating the correlation 

degree including: 

a step of preparing plural groups of 
camera sets each being a group of some viewpoints 
selected from the plural viewpoints; and 
25 a step of obtaining the correlation degree 

from the corresponding points on images included in 

each camera set, 

the step of determining the existence 

probability including: 
30 a step of calculating the existence 

probability based on the correlation degree of each 
projection point obtained for each camera set; and 

a step of determining the existence 
probability of each projection point by performing 
35 integrating processing for the existence probability 
determined for each camera set. 

In addition, the present invention can 
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be configured as an image generation method 
including: 

a step of obtaining images of an object 
taken by changing focusing distance ; 
5 a step of obtaining a virtual viewpoint 

that is a viewpoint from which the object appearing 
in the images is viewed; 

a step of obtaining a three-dimensional 
shape of the object based on the images; and 
10 a step of generating an image of the 

object viewed from the virtual viewpoint based on 
the obtained three-dimensional shape of the object, 

the step of obtaining the three- 
dimensional shape of the object including: 
15 a step 0 f setting projection planes having 

a multi-layered structure in a virtual three- 
dimensional space; 

a step of determining a reference 
viewpoint for obtaining the three-dimensional shape 

20 of the object; 

a step of determining color information or 
brightness information of projection points, that 
are points on the projection planes, based on color 
information or brightness information of 
25 corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining focusing degrees of 
the projection points based on focusing degrees of 
the corresponding points corresponding to the 
30 projection points; 

a step of calculating, for each of the 
projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that the object exists at a 
35 distance corresponding to a position of the 

projection point based on the focusing degree of 
each projection point; 
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the step of generating the image of the 
object viewed from the virtual viewpoint including: 

mixing color information or brightness 
information of the projection points overlapping 
5 when viewed from the virtual viewpoint in a ratio 
corresponding to the existence probability to 
determine color information or brightness 
information of each point of the image to be 
generated . 

10 In addition, the present invention can be 

configured as an image generation method including: 

a step of obtaining images of an object 
taken under different conditions; 

a step of obtaining a three-dimensional 
15 shape of the object based on the images; and 

a step of generating an image of the 
object viewed from a viewpoint of an observer based 
on the obtained three-dimensional shape of the 
object, 

20 the step of obtaining the three- 

dimensional shape of the object including: 

a step of setting projection planes having 
a multi-layered structure in a virtual three- 
dimensional space; 

25 a step of determining a reference 

viewpoint for obtaining the three-dimensional shape 
of the object; 

a step of determining color information or 
brightness information of projection points, that 

30 are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining, for each of the 

35 projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 
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at the projection point; 

the step of determining the existence 

probability including: 

a step of calculating an evaluation 
reference value of each projection point from image 
information of the corresponding points ; 

a step of performing statistical 
processing on the evaluation reference value of each 

projection point; and 

a step of calculating the existence 
probability of each projection point based on the 
evaluation reference value on which the statistical 
processing has been performed. 

In addition, the present invention can be 
15 configured as a three-dimensional image display 

method including: 

a step of obtaining images of an object 
taken under different conditions ; 

a step of obtaining a three-dimensional 
2 0 shape of the object based on the images ; 

a step of setting a viewpoint position 
from which an observer observes plural image display 
planes existing at different depth positions when 
viewed from the observer; 
25 a step of generating two-dimensional 

images to be displayed on each image display plane 
based on the obtained three-dimensional shape of the 

object; and 

a step of presenting a three-dimensional 

30 image of the object by displaying the generated two- 
dimensional images on each display plane, 

the step of obtaining the three- 
dimensional shape of the object including: 

a step of setting projection planes having 
35 a multi-layered structure in a virtual three- 
dimensional space; 

a step of determining a reference 
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viewpoint for obtaining the three-dimensional shape 
of the object; 

a step of determining color information or 
brightness information of projection points, that 
5 are points on the projection planes, based on color 
information or brightness information of 
corresponding points, on the obtained images, 
corresponding to the projection points; 

a step of determining, for each of the 
10 projection points overlapping when viewed from the 
reference viewpoint, an existence probability that 
is a probability that a surface of the object exists 
at the projection point; 

the step of generating the two-dimensional 
15 images including: 

converting the color information or the 
brightness information and the existence probability 
of the projection point into color information or 
brightness information and existence probability of 
20 each display point that is a point on the image 

display plane corresponding to the projection plane 
on which the projection point exists so as to 
generate the two dimensional images; 

the step of presenting the three- 
25 dimensional image of the object including: 

displaying the color information or 
brightness information on each display point with 
brightness according to the existence probability, 

30 EFFECT OF THE INVENTION 

According to the present invention, when 
generating the image of the object by obtaining the 
three-dimensional shape of the object from plural 
images, it becomes possible to reduce remarkable 

35 deterioration of image quality that occurs in a part 
where the reliability of estimation of the shape of 
the object is low. In addition, it becomes possible 
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to generate the image having small partial 
deterioration of image quality in a short time even 
by using an apparatus having low processing 
performance. Further, it becomes possible to 
5 downsize a taking apparatus for taking images used 
for obtaining the geometrical model of the object so 
as to simplify the apparatus configuration. 

BRIEF DESCRIPTION OF THE DRAWINGS 
10 Fig.l is a diagram for explaining problems 

a conventional virtual viewpoint image; 

Fig. 2 is a schematic diagram for 
explaining the principle of the virtual viewpoint 
image generation method of the first embodiment, and 
15 is a diagram showing examples of projection plane 

groupr camera, reference viewpoint, projection point 
and corresponding point; 

Fig.3A and Fig.3B are diagrams for 
explaining the principle of the virtual viewpoint 
20 image generation method of the first embodiment; 

Fig.4A and Fig.4B are diagrams for 
explaining the principle of the virtual viewpoint 
image generation method of the first embodiment, and 
are diagrams showing an example of mixing processing 
25 according to transparency of projection points; 

Fig. 5 is a schematic diagram for 
explaining the principle of the virtual viewpoint 
image generation method of the first embodiment, and 
is a diagram showing examples of object, projection 
30 plane group, reference viewpoint, virtual projection 
point and projection point; 

Fig. 6 is a schematic diagram showing an 
outline configuration of a virtual viewpoint image 
generation apparatus of an embodiment 1-1/ and is a 
35 block diagram showing a configuration of the inside 
of the image generation apparatus; 

Fig. 7 is a schematic diagram showing an 
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outline configuration of a virtual viewpoint image 
generation apparatus of an embodiment 1-1, and is a 
configuration example of a system using the image 
generation apparatus; 
5 " Fig. 8 is a schematic diagram for 

explaining the mathematical principle model of the 
virtual viewpoint image generation method using the 
virtual viewpoint image generation apparatus of the 
embodiment 1-1, and is a diagram showing an example 
10 of projection conversion; 

Fig. 9 is a schematic diagram for 
explaining the mathematical principle model of the 
virtual viewpoint image generation method using the 
virtual viewpoint image generation apparatus of the 
15 embodiment 1-1, and is a diagram showing an example 
of coordinates conversion; 

Fig. 10 is a schematic diagram for 
explaining the generation processing procedure for 
the virtual viewpoint image of the embodiment 1-1, 
and is a flow diagram showing the whole generation 



20 



processing; 

Fig. 11 is a schematic diagram for 
explaining the generation processing procedure for 
the virtual viewpoint image of the embodiment 1-1, 
25 and is a concrete flow diagram showing steps for 
generating the virtual viewpoint image; 

Fig. 12 is a schematic diagram for 
explaining the generation processing procedure for 
the virtual viewpoint image of the embodiment 1-1, 
and is a diagram showing an example of a setting 
method of the projection plane; 

Fig. 13 is a schematic diagram for 
explaining the generation processing procedure for 
the virtual viewpoint image of the embodiment 1-1, 
and is a diagram showing examples of projection 
points, projection point series and a set of 
projection point series; 
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Fig.14 is a schematic diagram for 
explaining the generation processing procedure for 
the virtual viewpoint image of the embodiment 1-1, 
and is a diagram showing examples of a reference 
5 viewpoint, projection points and angles based on 
various camera positions for explaining mixing 
processing of color information; 

Fig. 15 is a schematic diagram for 
explaining the generation processing procedure for 
10 the virtual viewpoint image of the embodiment 1-1, 

and is a diagram showing an example of corresponding 
point matching processing; 

Fig. 16 is a schematic diagram for 
explaining the generation processing procedure for 
15 the virtual viewpoint image of the embodiment 1-1, 
and is a diagram for explaining rendering 
processing; 

Fig .17 is a schematic diagram for 
explaining the generation processing procedure for 
20 the virtual viewpoint image of the embodiment 1-1, 
and is a diagram showing an example of a generated 
virtual viewpoint image; 

Fig- 18 is a schematic diagram showing an 
application example of a system to which the virtual 
25 viewpoint image generation apparatus of the 
embodiment 1-1 is applied; 

Fig.l9A is a flow diagram indicating 
processing that is a feature of the embodiment 1-2, 
and Fig.l9B is a flow diagram showing an example of 
30 concrete processing procedure of steps for 
determining transparency information; 

Fig ,20 is a schematic diagram for 
explaining the virtual viewpoint image generation 
method of the embodiment 1-3, and is a diagram 
35 showing examples of pro jection. plane group, 
reference viewpoint, virtual viewpoint and 
projection points; 
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Fig.21 is a schematic diagrams for 
explaining the principle of the image generation 
method of the second embodiment, and is a diagram 
for explaining a concept of a method for generating 
5 the image to be displayed; 

Fig.22 is a schematic diagram for 
explaining the principle of the image generation 
method of the second embodiment, and is a diagram 
representing Fig. 21 in two-dimensional; 
10 ~ Fig.23A and Fig.23B are diagrams for 

explaining a method for obtaining the correlation 
degree of the corresponding points ; 

Fig-24A and Fig.24B are diagrams for 
explaining a problem when obtaining the correlation 
15 degree of the corresponding points; 

Fig. 25 is a schematic diagram for 
explaining the principle of the image generation 
method of the second embodiment, and is a diagram 
for explaining a method for solving the problem when 
obtaining the correlation degree; 

Fig.26A and Fig.26B are diagrams for 
explaining an example of a method for improving 
accuracy of the existence probability 

Fig. 27 is a schematic diagram for 
25 explaining the principle of the image generation 
method of the second embodiment; 

Fig. 28 is a schematic diagram for 
explaining the principle of the image generation 
method of the second embodiment ; 

Fig. 29 is a schematic diagram for 
explaining an image generation method of the 
embodiment 2-1, and is a flow diagram showing an 
example of a general processing procedure; 

Fig. 30 is a schematic diagram for 
35 explaining an image generation method of the 

embodiment 2-1, and is a flow diagram showing an 
example of a processing procedure of steps for 
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determining color information and existence 
probability of projection points in Fig. 29; 

Fig. 31 is a schematic diagram for 
explaining an image generation method of the 
5 embodiment 2-1, and is a flow diagram showing an 
example of steps for determining the existence 
probability in Fig. 30; 

Fig.32 is a schematic diagram for 
explaining an image generation method of the 
10 embodiment 2-1 r and is a diagram showing an example 
foe setting a camera set; 

Fig .33 is a schematic diagram for 
explaining an image generation method of the 
embodiment 2-1, and is a diagram for explaining a 
15 method for converting information of the projection 
plane to information of display plane; 

Figs.34A and 34B are diagrams for 
explaining a method for converting information of 
the projection plane to information of display 
20 plane; 

Fig .35 is a block diagram showing a 
configuration example of the image generation 
apparatus to which the image generation method of 
the embodiment 2-1 is applied; 
25 Fig. 36 is a diagram showing a 

configuration example of an image display system 
using the image generation apparatus to which the 
image generation method of the embodiment 2-1 is 
applied; 

30 Fig. 37 is a diagram showing another 

configuration example of an image display system 
using the image generation apparatus to which the 
image generation method of the embodiment 2-1 is 
applied; 

35 Fig. 38 is a schematic diagram for 

explaining an image generation method of the 
embodiment 2-2, and is a flow diagram showing an 
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example of the whole processing procedure; 

Fig. 39 is a schematic diagram for 
explaining an image generation method of the 
embodiment 2-2, and is a diagram for explaining a 
5 principle of rendering ; 

Fig. 40 is a schematic diagram for 
explaining an image generation method of the 
embodiment 2-2, and is a diagram for explaining 
problems that arise in the image generation method 

10 in this embodiment ; 

Figs.41A and B are schematic diagrams for 
explaining an image generation method of the 
embodiment 2-2, and are diagrams for explaining a 
method for solving the problems that arise in the 
15 image generation method in this embodiment ; 

Fig. 42 is a schematic diagram for 
explaining an image generation method of the 
embodiment 2-2, and is a flow diagram showing an 
example of the processing procedure for converting 
20 the existence probability to transparency; 

Fig. 43 is a schematic diagram for 
explaining the principal of the image generation 
method of the third embodiment, and is a diagram 
showing a setting example of the projection planes 
25 and the reference viewpoint; 

Fig. 44 is a schematic diagram for 
explaining the principal of the image generation 
method of the third embodiment, and is a diagram 
showing a setting example of the projection planes 
30 and the reference viewpoint; 

Fig-45 is a schematic diagram for 
explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining a method for determining the color 
35 information and the focusing degree of the 
projection point; 

Fig- 46 is a schematic diagram for 
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explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining a method of determining the existence 
probability of the projection point; 

Fig. 47 is a schematic diagram for 
explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining a method of determining the existence 
probability of the projection point; 

Fig. 48 is a schematic diagram for 
explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining a method of determining the existence 
probability of the projection point; 
15 Fig. 49 is a schematic diagram for 

explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining a method for generating an image viewed 
from the virtual viewpoint; 
20 Fig. 50 is a schematic diagram for 

explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining problems in the image generation method 
in the present invention; 
25 Fig. 51 is a schematic diagram for 

explaining the principal of the image generation 
method of the third embodiment, and is a diagram for 
explaining a method for solving the problems in the 
image generation method in the present invention; 
30 " Fig. 52 is a schematic diagram for 

explaining the mathematical model of the image 
generation method of the third embodiment, and is a 
diagram showing relationship among projection points, 
corresponding points and points on the image to be 
35 generated; 

Fig. 53 is a schematic diagram for 
explaining the mathematical model of the image 
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generation method of the third embodiment, and is a 
diagram for explaining a method for converting 
points on the space to pixels on the image; 

Fig. 54 is a schematic diagram for 
5 explaining the image generation method of the 

embodiment 3-1, and is a a flow diagram showing a 
generation procedure for the image; 

Fig. 55 is a schematic diagram for 
explaining the image generation method of the 
10 embodiment 3-1, and is a diagram for explaining a 
method for setting the projection point series; 

Fig. 56 is a schematic diagram for 
explaining the image generation method of the 
embodiment 3-1, and is a flow diagram showing a 
15 concrete example of processing of the step 10305 of 
Fig. 54 ; 

Fig. 57 is a schematic diagram for 
explaining the image generation method of the 
embodiment 3-1, and is a diagram for explaining a 

20 rendering method; 

Fig. 58 is a schematic diagram showing a 
general configuration of the apparatus for 
generating the image using the image generation 
method of the embodiment 3-1, and is a a block 
25 diagram showing the configuration of the apparatus; 

Fig. 59 is a diagram for explaining a 
configuration example of the object image taking 
means the embodiment 3-1; 

Fig. 60 is a diagram for explaining a 
30 configuration example of the object image taking 
means the embodiment 3-1; 

Fig. 61 is a diagram for explaining a 
configuration example of the object image taking 
means the embodiment 3-1; 
35 Fig. 62 is a schematic diagram showing a 

general configuration of the image generation system 
using the image generation apparatus of the 
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embodiment 3-1, and is a diagram showing 
configuration example of the image generation 

system; , . _ 

Fig 63 is a schematic diagram showxng a 

general configuration of the image generation system 
using the image generation apparatus of the 
embodiment 3-1, and is a diagram showing another 
configuration example of the image generation 
system; 

Fig. 64 is a flow diagram showing 
processing of the virtual viewpoint image generation 
method of the embodiment 3-2; 

Fig 65 is a schematic diagram for 
explaining other generation method in the image 
generation method in the third embodiment; 

Fig. 66 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a flow diagram showing an 
example of the whole processing procedure; 

Fig. 67 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram showing an example 
of a method for setting the projection plane; 

Fig. 68 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram showing an example 
of a method for setting the projection plane; 

Fig. 69 is a schematic diagram for 
explaining the image generation method of the 
30 embodiment 4-1, and is a diagram for explaining a 
method for setting projection point series; 

Fig. 70 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a flow diagram showing an 
35 example of processing procedure of steps for 

determining the color information and the existence 
probability of the projection point ; 
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Fig.71 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram for explaining a 
method for determining the existence probability; 

Fig. 72 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram for explaining a 
method for determining the existence probability ; 
Fig.73 is a schematic diagram for 
, explaining the image generation method of the 

embodiment 4-1, and is a diagram for explaining a 
method for determining the existence probability. 

Fig. 74 is a schematic diagram for 
explaining the image generation method of the 
5 embodiment 4-1, and is a diagram for explaining a 
method for determining the existence probability, 

Fig. 7 5 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram for explaining a 
0 method for generating the two-dimensional images to 
be displayed on each image display plane- 
Fig. 7 6 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram for explaining a 
,5 method for generating the two-dimensional images to 
be displayed on each image display plane; 

Fig. 77 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-1, and is a diagram for explaining a 
30 method for generating the two-dimensional images to 
be displayed on each image display plane; 

Fig. 78 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-2, and is a diagram showing 
35 relationship between the projection point and the 
corresponding point; 

Fig. 79 is a schematic diagram for 
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explaining the image generation method of the 
embodiment 4-2 , and is a flow diagram showing an 
example of steps for determining the color 
information and the existence probability of the 
5 projection point; 

Fig -80 is a schematic diagram for 
explaining the image generation method of the 
embodiment 4-2, and is a diagram for explaining a 
method for obtaining the existence probability; 

10 Fig -81 is a schematic diagram for 

explaining the image generation method of the 
embodiment 4-2, and is a diagram for explaining a 
method for obtaining the existence probability; 

Fig.82 is a schematic diagram for 

15 explaining an arbitrary viewpoint image generation 
method of the embodiment 4-3, and is a flow diagram 
showing an example of the whole processing 
procedure; 

Fig. 83 is a schematic diagram for 
20 explaining an arbitrary viewpoint image generation 
method of the embodiment 4-3, and is a diagram for 
explaining the principle of rendering; 

Figs,8 4A and 84B are flow diagrams showing 
processing procedure for converting the existence 
25 probability to the transparency 

Fig , 85 is a schematic diagram showing a 
general configuration of the image generation 
apparatus of the embodiment 4-4; 

Fig. 8 6 is a schematic diagram showing a 
30 general configuration of the image generation 
apparatus of the embodiment 4-4; 

Fig. 87 is a schematic diagram showing a 
general configuration of the image generation 
apparatus of the embodiment 4-4, and is a diagram 
35 showing a configuration example of the image 
generation system using the image generation 
apparatus; 
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Fig.88 is a schematic diagram showing a 
general configuration of the image generation 
apparatus of the embodiment 4-4, and is a diagram 
showing a configuration example of the image 
5 generation system using the image generation 
apparatus; 

Fig. 89 is a schematic diagram showing a 
general configuration of the image generation 
apparatus of the embodiment 4-4, and is a diagram 
10 showing a configuration example of the image 
generation system using the image generation 
apparatus ; 

Fig. 90 is a schematic diagram for 
explaining the three-dimensional image display 
15 method of the embodiment 5-1 according to the fifth 
embodiment, and is a flow diagram showing an example 
of the whole processing procedure; 

Fig. 91 is a schematic diagram for 
explaining the three-dimensional image display 
20 method of the embodiment 5-1, and is a diagram 
showing an example of a method for setting the 

projection plane; 

Fig. 92 is a schematic diagram for 
explaining the three-dimensional image display 
25 method of the embodiment 5-1, and is a diagram 
showing an example of a method for setting the 

projection plane; 

Fig . 93 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1. and is a diagram for 
explaining a method for setting projection points; 

Fig. 94 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1, and is a flow diagram 
35 showing an example of processing procedure of steps 
for determining the color information and the 
existence probability of the projection point; 
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Fig. 95 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1, and is a diagram for 
explaining a method for determining the existence 

5 probability; 

Fig. 96 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1, and is a diagram for 
explaining a method for determining the existence 

10 probability; 

Fig. 97 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1, and is a diagram for 
explaining a method for determining the existence 

15 probability; 

Fig. 98 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1, and is a diagram for 
explaining a method for generating the two- 
2 0 dimensional images to be displayed on each image 
display plane; 

Fig. 99 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-1, and is a diagram for 
25 explaining a method for generating the two- 
dimensional images to be displayed on each image 
display plane; 

Fig. 100 is a schematic diagram for 
explaining the three-dimensional image display 
30 method of the embodiment 5-1, and is a diagram for 
explaining a method for generating the two- 
dimensional images to be displayed on each image 

display plane; 

Fig. 101 is a schematic diagram for 
35 explaining the three-dimensional image display 
method of the embodiment 5-2, and is a diagram 
showing relationship between the projection point 
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and the corresponding point; 

Fig. 102 is a schematic diagram for 
explaining the three-dimensional image display 
method of the embodiment 5-2, and is a flow diagram 
5 showing an example of steps for determining the 

color information and the existence probability of 
the projection point; 

Fig. 103 is a schematic diagram for 
explaining the three-dimensional image display 
10 method of the embodiment 5-2, and is a diagram for 
explaining a method for obtaining the existence 

probability; 

Fig- 104 is a schematic diagram for 
explaining the three-dimensional image display 
15 method of the embodiment 5-2, and is a diagram for 
explaining a method for obtaining the existence 
probability. 

[Explanation of symbols] 
20 (First embodiment) 

1,1A,1B,1C ... virtual viewpoint image generation 
apparatus, 101 ... virtual viewpoint determination 
means, 102 ... object image obtaining means, 103 ... 
image generation means, 103a ... projection plane 
25 determination means, 103b ... reference viewpoint 

determination means, 103c ... texture array keeping 
means, 103d ... corresponding point matching 
processing means, I03e ... color information 
determination means, 103f ... existence probability 
30 information determination means, 103g ... rendering 
means, 104 ... generated image output means, 2 ... 
viewpoint position input means, 3 ... object taking 
means (camera), 4 ... image display means, 6 ... virtual 
viewpoint image, 7 ... image of object, 7 A ... a part 
35 where the image is deteriorated, 7B ... a part where 
the image is lost 

(Second embodiment) 
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6,6A,6B,6C - image generation apparatus, 601 ... 
object image obtaining means, 602 ... observer 
viewpoint setting means, 603 ... projection plane 
setting means, 604 ... projection plane information 
5 storing area keeping means, 605 ... color 

information/existence probability determination 
means, 606 ... projection plane information - display 
plane information conversion means, 607 ... image 
output means, 7,7 A, 7B ... image display means, 8,8A,8B 
10 ... object image taking means, 9,9A,9B ... reference 
viewpoint input means 

(Third embodiment) 
2,2A,2B,2C ... image generation apparatus, 201 ... 
object image obtaining means, 202 ... virtual 
15 viewpoint setting means, 203 ... projection plane etc. 
setting means, 204 ... texture array keeping means, 
205 ... color information/existence probability 
determination means, 206 ... rendering means, 207 ... 
generated image output means, 3,3A,3B ... object image 
20 taking means, 4,4A,4B ... viewpoint information input 
means, 5,5A,5B ... image display means, 6 ... polarizing 
binary optical system, 7,7A,7B ... image sensor, 8 ... 
beam splitter, 9 ... polarizing filter, 10 ... variable 
focus lens, 11a, lib, 11c, lid ... fixed focus lens, 12 ... 
2 5 lens holder 

(Fourth embodiment) 
2,2A,2B,2C ... image generation apparatus, 201 ... 
object image obtaining means, 202 ... observer 
viewpoint setting means, 203 ... projection plane etc, 
30 setting means, 204 ... texture array keeping means, 
205 ... color information/existence probability 
determination means, 20 6 ... projection plane 
information - display plane information conversion 
means, 207 ... image output means, 208 ... rendering 
35 means, 3,3A,3B ... image display means, 4,4A,4B ... 
object image taking means, 5,5A,5B ... reference 
viewpoint information input means 
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( Fifth erobod intent) 
2 2A,2B,2C ... three-dimensional image generation 
apparatus, 201 ... object image obtaining means, 202 ... 
observer viewpoint setting means, 203 ... projection 

5 plane etc. setting means, 204 ... texture array 

keeping means, 205 ... color information/existence 
probability determination means, 206 ... projection 
plane information - display plane information 
conversion means, 207 ... image output means, 3,3A,3B 

10 ... image display means, 4,4A,4B ... object image taking 
means, 5,5A,5B ... reference viewpoint input means 

PREFERRED EMBODIME N T S FOR CARRY IN G OUT THE INVENTION 
In the following, first to fifth 
15 embodiments are described as the preferred 
embodiments for carrying out the invention. 
[First embodiment] 

First, the first embodiment of the present 
invention is described. The first embodiment 
20 mainly corresponds to claims 1-11. In this 

embodiment, although three primary colors of red (R) , 
green (G) and blue (B) are used for representing 
color information as an example, other 
representation such as brightness <Y> or color 
25 difference (U, V) can be used. In addition, for 
~~ black-and-white images, only brightness information 
can be used as the color information. In figures 
for explaining the first embodiment, the same 
reference signs are assigned for the same functions. 
30 Before explaining each embodiment of the 

~ " first embodiment, principle of a virtual viewpoint 
image generation method of the first embodiment is 
described. 

Figs. 2-5 are schematic diagrams for 
35 explaining the principle of the virtual viewpoint 
image generation method of the present invention. 
Fig. 2 shows examples of projection plane group, 
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camera, reference viewpoint, projection point and 
corresponding point. Fig.3A and Fig.3B show 
examples of graphs of correlation degree between 
corresponding points, Fig.4A shows an example of 
5 mixing processing according to the transparency of 
projection points, and Fig.4B represents mixing 
processing for color information according to 
transparency in a color space. Fig -5 shows examples 
of an object, a projection plane group, a reference 
10 viewpoint, a virtual viewpoint and a projection 
point - 

The method for generating the virtual 
viewpoint image in this invention includes: a step 1 
of setting the projection plane group having the 
15 multi-layered structure; a step 2 of obtaining 
points (corresponding points), on plural images 
taken by the camera, corresponding to each point 
(projection point) on the projection planes; a step 
3 of mixing color information of the plural 
20 corresponding points or selecting one of them to 

determine color information of the projection point; 
a step 4 of calculating, based on correlation 
degrees of the corresponding points or the adjacent 
regions, degree (existence probability information) 
25 of probability that the object exists at the 

distance of the projection point for each of the . 
plural projection points overlapping when viewed 
from a viewpoint (reference viewpoint) in a space; a 
step 5 of performing mixing processing on the color 
30 information of reference points overlapping when 

viewed from the virtual viewpoint according to the 
existence probability information so as to determine 
color information of each pixel in the virtual 
viewpoint; and a step 6 of repeating the steps 1-5 
35 for every point corresponding to pixels of virtual 

viewpoint image. That is, the method of the present 
invention does not intend to obtain a geometrical 
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object model accurate for every case and every 
portions like the conventional method. But, 
assuming that an estimated value having adequate 
reliability cannot be obtained in the distance 
estimation depending on picture-taking conditions 
a nd portions of the object, a portion, by which an 
estimated value of low reliability is obtained, is 
drawn vaguely so as to provide the portion with low 
contribution to image generation for avoiding 
extreme image deterioration, and a portion, by which 
distance data of high reliability is obtained, is 
drawn clearly so as to provide the portion with high 
contribution to image generation. 

The reliability of estimation is 
15 determined in the following way according to the 
degree of correlation (correlation degree) of 
corresponding points of the taken images. For 
example, as shown in Fig. 2, a reference viewpoint R, 
center positions (i-1, 2, ...,N) of the camera, and 
20 projection planes L 3 ( 2,...,M) that are parallel 
each other are set, and it is assumed that a 
corresponding point obtained by taking a projection 
point T 3 by a camera at a center position Ci is G i5 . 
As a result, for example, a set 
25 {G ia |i=l,2,...,N} of corresponding points with respect 
to a projection point T„ on a projection plane L B is 
obtained, and the degree of correlation (correlation 
degree) of them can be calculated. 

A graph shown in Fig.3A or Fig.3B can be 
obtained by calculating the correlation degrees for 
plural projection points T 5 ( j-1, 2, M) that lie on a 
straight line when viewed from the reference 
viewpoint R, and by indicating the distance 1 
between the reference viewpoint R and the projection 
plane using the lateral axis, and indicating the 
correlation degree using the horizontal axis. A 
concrete calculation method of the correlation 
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degree is described in later-described embodiments. 
Here, the graph is shown assuming that the larger 
the correlation degree is, the larger the degree of 
the correlation among correspondence points is. 
5 At a distance corresponding to high 

correlation degree, corresponding points in plural 
cameras are similar so that there is a high 
possibility that the same point of the object is 
taken at the position of the projection point by the 
10 cameras. Therefore, there is a high possibility 

that the object exists at the distance. When only 
one distance by which the object exits on the line 
passing through the reference viewpoint R exists, it 
can be estimated that the object exists at a 
15 distance 1=1* corresponding to the highest 

correlation degree as shown in Fig.3A and Fig . 3B 

At this time, as shown in Fig.3A, when the 
correlation degree of the corresponding points at 
the distance 1=1* is far higher than other 
candidates, highly reliable estimation can be 
performed. However, as shown in Fig.3B, when there 
are plural candidates of estimation values such as 
the distance 1=1* and distance 1-1' and when the 
correlation degrees of the corresponding points are 
almost the same, the reliability of the estimation 



20 



25 



becomes low. 

In the case shown in Fig.3B, when a method 

for rendering only one projection point 
corresponding to the distance 1=1* of the highest 

30 correlation degree is adopted, large noise appears 
on a generated image if the object actually exists 
at the distance 1-1' due to estimation error. 

On the other hand, according to the 
present invention, a probability (existence 

35 probability information) that the object exists is 
calculated based on the correlation degree, so that 
plural projection points are rendered with clarity 
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according to the existence probability information. 
As a result, plural projection points are rendered 
vaguely when the estimation reliability is low, so 
that there is an effect that noise is inconspicuous 
5 on the generated image and an image looks better for 
the observer can be generated. 

On the other hand, a projection point 
corresponding to high existence probability is 
clearly rendered when the estimation reliability is 
10 high, so that a better image can be generated. 

In addition, the rendering method of 
this invention can be simply implemented using 
texture-mapping that is a basic method in computer 
graphics, so that there is an effect that computing 
15 workload can be decreased such that even three- 
dimensional graphics hardware mounted on a popular 
personal computer can perform rendering processing 
nicely. 

In addition, in the virtual viewpoint 
20 image generation method of this invention, each 
reference point on the projection plane has 
transparency with a gradation from transparent to 
opaque, and the transparency in each reference point 
is calculated by converting the existence 

25 probability information obtained by the step 4. In 
step 5, the mixing processing for obtaining color 
information of each point in the virtual viewpoint 
is successively performed from a projection point 
far from the virtual viewpoint to a projection point 

30 near to the virtual viewpoint. Color information 

obtained by the mixing processing up to a projection 
point is obtained by internally dividing, with a 
ratio according to transparency , between color 
information at the projection point and color 

35 information obtained by mixing processing up to a 
previous projection point. The color information 
obtained by the mixing processing is an internal 
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division of color information at a stage and next 
color information . 

For example, as shown in Fig.4A, 
considering a case in which projection planes 
5 L-j ( j^l, 2, M) r projection points Tj ( j =1 , 2 , M) and 
vectors Kj ( j = l , 2 , M) having elements of red, green 
and blue (R,G,B) and indicating color information 
are set in a color space represented by the 
following equation 1. 
10 [Equation 1] 

Kj e V 9 V = {{R,G,B)\0< R <1, 0<G<l,0<£<n} 

In addition, it is assumed that transparency 
otj ( j=l, 2,.„,M) of the projection point is set as 
15 shown in the following equation 2. 
[Equation 2] 

0 < a j < 1 

At this time, color information D ro obtained by 
mixing processing up to j=m can be represented as 

20 recurrence formulas shown in the following equation 
3 and equation 4, color information Dm obtained by 
performing mixing processing up to j=M that is 
nearest to the virtual viewpoint becomes color 
information at the virtual viewpoint. 

25 [Equation 3] 

[Equation 4] 

D x =a } K x 

At this time, as represented by the relationship of 
30 the equation 2 and the equation 3, the color 
information D m is an internally divided point 
between K m and D m _i in a color space V. Thus, as 
shown in Fig.4B, when K m , D m -i^v is satisfied, D p ev 



2005fl2J116B 21«5fl 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 47 



-37- 

holds true. 

Then f if conditions of the equation 1 and 
the equation 2 are satisfied, it is ensured that the 
following equation 5 holds true with respect to the 
5 color information Dm in the virtual viewpoint. 
[Equation 5] 

The equation 5 can be proved using mathematical 
induction, but detailed description is not provided 
10 here* 

That is, by setting the color information 
and the transparency at the projection point such 
that the equations 1 and 2 are satisfied, the color 
information of the virtual viewpoint can be limited 

15 within a proper color space v without fail. 

According to the above-mentioned features, 
when plural virtual viewpoint images for the same 
object are generated, even when color information 
and transparency of projection points are calculated 

20 from one reference viewpoint, if the color 

information and the transparency satisfy the 
equation 1 and the equation 2, images can be 
generated with proper color information bounds for 
all virtual viewpoint images. 

25 For example, as shown in Fig- 5, a case is 

considered in which, when an object Obj exists, and 
two projection planes Li and L 2 , a reference 
viewpoint R and a virtual viewpoint P are set, 
pieces of color information of projection points 

30 Ti, T 2 ,Ti' , T 2 ' are K lf K 2 , Ki' , K 2 f respectively, and 

degrees of probability of existence of object are 
Pi, p2/ Pi' r P2' respectively. 

The degree (existence probability 
information) of probability of existence of the 

35 object is calculated on a line passing through the 
virtual viewpoint R, and a sum of existence 
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probability information at projection points on the 
same straight line is 1. Since the surface of the 
object exists near the projection points T a ' and T 2 , 
existence probability information at the points are 
5 higher than those of T x and T 2 ' . Then, the 

existence probability information is represented by 
the following equations 6 and 7. 
[Equation 6] 

A so, A si 

10 [Equation 7] 

P[ s 1, #s0 

At this time, color information K A at a 
point A of the image plane of the virtual viewpoint 
P can be calculated by assigning weights to color 
15 information of projection points on the line PA and 
adding the weighted color information of the 
projection points as represented by the following 
equation 8 - 
[Equation 8] 

The equation 8 can be represented as the following 
equation 9 according to the equations 6 and 7 . 
[Equation 91 

K A K[ + K 2 

25 When being viewed from the virtual viewpoint P, TV 
is blocked by T». Therefore, brightness of each 
component of (R, G, B) is increased by Ki' as shown in 
the equation 9 although actual color information at 

the point A is K A =K2- 

In addition, when each component of K A and 
K A ' has large brightness, K A exceeds effective region 
of the color space. Therefore, clipping processing 
becomes necessary for K A to fall within a 
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predetermined limit . 

Accordingly, for example, results shown in 
equations 10 and 11 are obtained by calculating 
transparency from existence probability information 
5 using a calculation method described in an after- 
mentioned example 1-2. 
[Equation 10] 

a 2 = J3 2 , a x = 1 

[Equation 11] 

In the equation 10 and the equation 11, each of ot lr 
a 2 , olx* and a 2 ' is transparency at T x , T 2 , Ti' and T 2 ' 
respectively - 

For obtaining color information of each 

15 point with respect to the virtual viewpoint, mixing 
processing is performed successively from a 
projection point far from the virtual viewpoint 
toward a projection point near the projection point, 
and color information obtained by mixing processing 

20 up to a projection point is obtained by performing 
interior division between color information at the 
projection point and color information obtained by 
mixing processing up to a previous projection point 
at a ratio of transparencies r so that K A is 

25 represented by the following equation 12, 
[Equation 12] 

The equation 12 can be represented as the following 
equation 13 according to the equation 6, the 
30 equation 7, the equation 10 and the equation 11. 
The equation 13 represents good approximation of 
actual color information. 
[Equation 13] 
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As mentioned above, when performing image generation 
using the existence probability information as it is, 
there is no problem if the reference viewpoint and 
5 the virtual viewpoint are the same. But, when they 
are different, brightness may increase near blocked 
region. On the other hand, according to image 
generation in which the existence probability 
information is changed to transparency, the above 
10 mentioned phenomenon can be prevented. 

in addition, when the reference viewpoint 
and the virtual viewpoint are different in the image 
generation using the existence probability 
information as it is, there is no guarantee that 
15 color information that is calculated using after- 
mentioned equations falls within an effective color 
information limit, and correction processing is 
necessary, for example. On the other hand, 
according to the image generation in which the 
existence probability information is converted to 
the transparency, such correction is not necessary. 

In addition, according to the image 
generation in which the existence probability 
information is converted to transparency, a 
25 semitransparent object can be efficiently 

represented, so that there is an effect that the 
present invention can be applied more widely to more 
objects in the actual world. 

In addition, according to the virtual 
30 viewpoint image generation method, a group of 

projection planes specific for each camera is set in 
step 1. In step 3, as to color information of a 
projection point, color information of corresponding 
point of an image taken by a camera specific for a 
35 projection plane to which the projection point 
belongs is used. In step 4, the existence 
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probability information is calculated using a 
viewpoint of the camera, as the reference viewpoint, 
specific to the projection plane to which the 
projection point belongs. In mixing processing of 
color information of the virtual viewpoint in step 5, 
correction is performed according to position 
relationship between the virtual viewpoint and each 
reference viewpoint. Accordingly, since the group 
of the projection planes specific to each camera is 
set irrespective of position relationship among 
cameras, even if placement of the cameras is 
complicated or random, processing for setting the 
group of the projection planes is not affected so 
that image generation can be performed by consistent 

15 processing method. 

When the group of the projection planes 
specific to each camera is set, mixing processing 
between images taken by corresponding cameras is not 
necessary as to color information of the projection 

20 plane. Accordingly, for example, parallel 

processing can be performed when using a computer, 
so that the speed for generating the image can be 
increased. 

In addition, since all pieces of color 
25 information of the group of the projection planes 
associated with the same camera are the same, a 
texture memory for storing the color information can 
be shared when performing processing by a computer. 
Therefore, even if the number of projection planes 
30 is large, the memory is not consumed very much, so 
that workload in the apparatus used for image 
generation can be decreased. 

In addition, since the camera 
corresponding to a projection plane is uniquely 
35 determined, calibration such as correction of 

distortion of a lens can be performed easily and at 
high speed by setting corresponding relationship of 
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coordinates of them. 

Therefore, a program that causes a 
specific apparatus or a widespread personal computer 
or the like to perform the virtual viewpoint image 
generation method of the first embodiment of the 
present invention has wide application ranges and 

high versatility. 

In the following, an apparatus for 
performing the virtual viewpoint image generation 
method of the first embodiment and a concrete image 
generation method are described by taking examples. 
(Embodiment 1-1) 

Figs. 6 and 7 are a schematic diagram 
showing an outline configuration of a virtual 
15 viewpoint image generation apparatus of the 

embodiment 1-1 of the present invention- Fig. 6 is a 
block diagram showing a configuration of the inside 
of an image generation apparatus, and Fig. 7 is a 
configuration example of a system using the image 

20 generation apparatus. 

In Fig. 6, 1 indicates a virtual viewpoint 
image generation apparatus, 101 indicates a virtual 
viewpoint determination means, 102 indicates an 
object image obtaining means, 103 indicates an image 
generation means, 103a indicates a projection plane 
determination means, 103b indicates a reference 
viewpoint determination means, 103c indicates a 
texture array keeping means, 103d indicates a 
corresponding point matching processing means, I03e 
indicates a color information determination means, 
103f indicates an existence probability information 
determination means, I03g indicates a rendering 
means, 104 indicates a generated image output means, 
2 indicates a viewpoint position input means, 3 
35 indicates an object taking means, 4 indicates an 

image display means. In addition, in Fig. 7, "User- 
indicates a user of the virtual viewpoint image 



25 



30 



2005S12E16B 218§36# 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 53 



-43- 



gene rat!on apparatus, and «Obj" indicates an object. 

As shown in Figs . 6 and 1, the virtual 
viewpoint image generation apparatus 1 of the 
embodiment 1-1 includes the virtual viewpoint 
5 determination means 101 for determining parameters 
of a viewpoint (virtual viewpoint) input by using 
the viewpoint position input means 2 by the user 
-User", the object image obtaining means 102 for 
obtaining images of the object Obj taken by the 
10 object taking means (camera) 3, an image generation 
means 103 for generating, based on the obtained 
images of the object Obj, an image (virtual 
viewpoint image) of the object Obj viewed from the 
virtual viewpoint, and the generated image output 
15 means 104 for displaying the virtual viewpoint image 
generated by the image generation means 103 on the 
image display means 4. 

The virtual viewpoint determination means 
101 determines position, direction and angle of view, 
20 for example, as the parameters of the virtual 

viewpoint. The viewpoint position input means 2 may 
be a device such as a mouse for performing selection 
based on operation of the user User, for example as 
shown in Fig. 7, or a device such as a keyboard by 
25 which the user User directly inputs numerals, or may 
be a position/posture detection sensor mounted by 
the user User. In addition, the means may be 
provided by other program or a program provided via 
a network. 

30 The object image obtaining means 102 can 

successively obtain the position/posture of the 
object that changes every moment at predetermined 
intervals, for example, at 30Hz, and also can obtain 
a still image of the object at any time, and also 

35 can obtain the image by reading out the object image 
taken beforehand from a recording apparatus. By the 
way, it is desirable that the object images viewed 
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from the plural viewpoint positions are taken at the 
same time by synchronizing all cameras. But, this 
condition is not necessary if changes of the 
position and the posture of the object are 
5 sufficiently slow so that the object can be regarded 
as a still object. 

As shown in Fig. 6, the image generation 
means 103 includes projection plane determination 
means 103a for determining positions and shapes of 
10 the projection planes used for image generation, the 
reference viewpoint determination means 103b for 
determining the position of the reference viewpoint, 
the texture array keeping means 103c for assigning 
an array of texture images, to be mapped to the 
15 projection planes, to the memory, the corresponding 

point matching processing means 103d for associating, 
with each other, parts where the same region of the 
object appears in the images of the object obtained 
by the object image obtaining means 102 of plural 
20 viewpoint positions, the color information 

determination means 103e for determining color 
information in the texture array kept by the 
texture array keeping means 103c by performing 
mixing processing on color information of the 
25 obtained plural images of the object, the existence 
probability information determination means 103f for 
determining degrees (existence probability 
information) of probability that the object exists 
on the projection planes in the texture array kept 
30 by the texture array keeping means 103c based on the 
result of the corresponding point matching 
processing means 103d, and the rendering means 103g 
for rendering the projection planes viewed from the 
virtual viewpoint based on the color information 
35 determined by the color information determining 

means 103e and the existence probability information 
determined by the existence probability information 
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determining means 103f. 

The array kept in the texture array 
keeping means 103c includes color information and 
existence probability information of each pixel, m 
which three primary colors of red (R) , green (G) and 
blue (B) , and the existence probability information 
are represented by 8 bits respectively, for example. 
However, the present invention is not dependent on 
such particular data representing format. 

The image display means 4 is a display 
apparatus such as CRT (cathode Ray Tube), LCD 

(Liquid Crystal Display) and PDP (Plasma Display 
Panel) connected to the generated image output means 
104 The image display means 4 may be a display 
apparatus of a two-dimensional plane shape, or may 
be a display apparatus having a curved surface 
surrounding the user User. When a display apparatus 
that can display a three-dimensional shape is used 
as the image display means 4, the virtual viewpoint 
determination means 101 determines two virtual 
viewpoints corresponding to the user' s right and 
left eyes and the image generation means 103 
generates virtual viewpoint images viewed from the 
two virtual viewpoints so that independent images 
can be provided to the right and left eyes of the 
user- in addition, by generating images viewed from 
more than two virtual viewpoints and using a three- 
dimensional display that can display images haying 
more than two parallaxes, the three-dimensional 
30 image can be provided to equal to or more than one 

in addition, the system using the virtual 
viewpoint image generation apparatus 1 has a 
configuration shown in Fig. 7, for example. When the 
35 user User specifies desired viewpoint position / 
direction / image angle to the virtual viewpoint 
image generation apparatus 1 via the virtual 
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position input means 2, the virtual viewpoint image 
generation apparatus 1 photographs the object Obj 
with the object taking means (cameras) 3 so as to 
obtain the images. After that, the virtual 
5 viewpoint image generation apparatus 1 generates an 
image (virtual viewpoint image) from the specified 
viewpoint based on the obtained images of the object. 
The obtained virtual viewpoint image is presented to 
the user User using the image display means 4. 
XO The system configuration shown in Fig -7 

shows an implementation example of the image 
generation apparatus in the present invention. The 
claims of the present invention are not limited to 
such configuration, and positions, shapes and 
15 implementation of each apparatus can be freely 

determined within the scope of the present invention. 

In the following, before describing 
concrete processing procedure of the image 
generation processing by the image generation means 
20 103, a mathematical principle model on which the 
processing is based is described. 

Figs. 8 and 9 are schematic diagrams for 
explaining the mathematical principle model of the 
virtual viewpoint image generation method using the 
25 virtual viewpoint image generation apparatus of this 
embodiment 1-1. Fig. 8 shows an example of 
projection conversion, and Figs ♦ 9 show an example of 
coordinates conversion. 

In the image generation processing using 
30 the virtual viewpoint image generation apparatus of 
this embodiment 1-1, as shown in Fig. 8, for example, 
center positions C± ( i=l , 2 , — , N ) of the cameras 3, a 
virtual viewpoint P and projection planes 
L 3 { j=l, 2,,..,M) are set. In the following, it is 
35 assumed that the center position Ci of the camera 

indicates the camera itself in order to identify the 
plural cameras 3, and that, in the same way, P 
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indicates the virtual viewpoint itself and indicates 
the center position of the virtual viewpoint. 

Although the cameras Ci are placed in a 
lateral line in Fig. 8, the present invention is not 
limited to such placement. For example, various 
placements such as two-dimensional lattice, arc- 
shape and the like can be applied. In addition, the 
placement of the projection planes Lj are not 
necessarily parallel. Curved planes can be used as 
described later in the embodiment 1-3- However, in 
the descriptions of this embodiment 1-1, the 
projection planes L 3 are plane- 

In the virtual viewpoint image generation 
method of the present embodiment, in order to 
15 generate an image of the virtual viewpoint P at 

which the camera is not placed actually based on the 
images of the object Obj obtained at the points at 
which the cameras are actually placed, a procedure 
is basically adopted in which a part of the image of 
20 the object taken by the camera Ci is texture-mapped 
on the projection plane Lj that is assumed to exist 
in the virtual viewpoint image generation apparatus 
1 such as a computer, then, an image obtained by 
viewing the texture-mapped projection planes from 
25 the virtual viewpoint P is generated by coordinate 
calculation processing. 

When performing such processing, each of 
three-dimensional points of the virtual viewpoint P 
and the cameras C t is projected to a two-dimensional 
30 point of each image plane. 

Generally, a matrix for projecting a point 
(X,Y,Z) in a three-dimensional space to a point 
(x,y) in an image plane is a matrix having three 
rows and four columns, and can be represented by the 
35 following equation 14 and the equation 15. 
[Equation 14] 
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4>21 

In this case, for example, a matrix <D 0 representing 
perspective projection conversion of focal length f 
in which the original point is the center is as 
shown in the following equation 16- 
[Equation 16] 



10 An image dealt by the computer is a so-called 
digital image, and is represented by a two- 
dimensional array in a memory. The coordinate 
system (u,v) representing the positions of the array 
is called a digital image coordinate system. 

15 At this time, for example, one point on a 

digital image having a size of 640 pixels * 480 
pixels can be represented by a variable u that takes 
one of integers from 0 to 639 and a variable v that 
takes one of integers from 0 to 47 9, and color 

20 information at the point is represented by data 

obtained by quantizing red (R) , green (G) and blue 
(B) information at the address with 8 bits, for 
example , 
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In addition, at this time, the image 
coordinates (x,y) shown in Fig.9A are associated 
with the digital image coordinates (u,v) shown in 
Fig.9B in an one-to-one correspondence manner, and 
they have the following relationship shows in the 
equation 17. 



25 



[Equation 17] 
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L 1 , 
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-k u cot0 
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The x axis shown in Fig.9A is parallel to 
10 the u axis shown in Fig.9B, and it is assumed that 
unit lengths for the u axis and the v axis are k u 
and k v respectively, and an angle between the u axis 
and the v axis is 9.,. 

Although the digital image coordinates (u, 
15 v) take discrete values when writing into or reading 
out the two-dimensional array, it is assumed that 
the coordinates take continuous values unless 
otherwise specified in the following descriptions, 
and that proper discretization processing is 
20 performed when accessing the array. 

in addition to the coordinate conversion 
shown in the relationship of the equation 17, 
conversion for correcting distortion by aberration 
of the lens can be performed. 

In the following, a concrete procedure for 
generating the virtual viewpoint image is described 
using the mathematical principle model. 

Figs. 10-17 are schematic diagrams for 
explaining the generation processing procedure for 
30 the virtual viewpoint image of this embodiment 1-1- 
Fig. 10 is a flow diagram showing the whole 
generation processing. Fig, 11 shows a concrete flow 
diagram showing steps for generating the virtual 
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viewpoint image. Fig. 12 shows an example of a 
setting method of the projection planes. Fig. 13 
shows examples of projection points, projection 
point series and a set of projection point series. 
5 Fig. 14 shows examples of a reference viewpoint, 

projection points and angles based on various camera 
positions for explaining mixing processing of color 
information. Fig. 15 shows an example of 
corresponding point matching processing. Fig. 16 is 
10 a figure for explaining rendering processing. 
Fig-1"7 shows an example of a generated virtual 
viewpoint image. 

As shown in Fig. 10, when generating the 
virtual viewpoint image using the virtual viewpoint 
15 image generation apparatus 1, first, the virtual 
viewpoint determination means determines the 
parameters of the virtual viewpoint P based on a 
request from the user User (step 501) . In the step 
501, the position, the direction and image angle and 
20 the like are determined, for example. 

Next, image of the object obj taken by the 
cameras 3 (Ci) are obtained by the object image 
obtaining means 102 (step 502) . 

Next, based on images of the object 
25 obtained by the object image obtaining means 102, an 
image (virtual viewpoint image) of the object Obj 
viewed from the virtual viewpoint P is generated 

(step 503) . 

In the step 503, for example, processing 
30 of each step shown in Fig. 11 is performed so that 
the virtual viewpoint image is generated. 

In the processing in step 503, the 
projection plane determination means 103a determines 
the position and the shape of the projection planes 
35 L 3 (jej, js{l,2,...,M}) of multiple-layered structure 
used for generating the virtual viewpoint image 
(step 503a) . When setting the projection planes Lj 
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in the step 503a, projection planes having plane 
shape are placed in parallel at the same intervals 
as shown in Fig. 8, for example. 

When the cameras Ci are arranged at the 
5 same intervals, the planes (projection planes) can 
be arranged at a series l d (d=l , 2 , 3, ...) of distances 
obtained by the following equation 18, wherein the 
placement interval of the camera is B, focal length 
of the camera is F, and the size of one pixel on the 

10 image is 5. 

[Equation 18] 

In this case, depth resolution of corresponding 
point matching among cameras is the same as the 

15 placement interval of the projection planes. That 

is, as shown in Fig. 12, assuming that cameras C n and 
C n -i are placed at an interval B, a point of the 
camera C„ on the image plane is A, a point 
corresponding to the point A in C n on the image 

20 plane of the camera C n -! is A 0 ' , end that a point of 
d-th pixels from A 0 ' is A d ' , corresponding points of 
the point A is a series of {Ao' I d=l, 2,...} and series 
of distances calculated in this case are provided by 

the equation 18- 
25 The setting example of the projection 

planes Lj is one example, and it is basically 
sufficient to set more than one different projection 
planes in the image generation method of the present 
invention. The present invention is not limited to 
30 such particular setting method of the projection 
planes. 

When the processing in step 503a ends, 
next, the reference viewpoint determination means 
103b determines a reference point (reference 
35 viewpoint) R used for calculating the degree 
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(existence probability information) of probability , 
of existence of the object on the projection point 
(step 503b) . The position of the reference 
viewpoint R may be the same as the position of the 
5 virtual viewpoint P, or, when there are plural 

virtual viewpoints, the position of the reference 
viewpoint R may be a barycenter position of the 
plural virtual viewpoints. But, the present 
invention does not provide a method dependent on how 
10 to decide a particular reference viewpoint. 

After performing the processing of the 
step 503b, next, many projection points are set on 
the projection planes (step 503c). At this time, 
the projection points are set such that the 
15 projection points exist on plural lines that pass 
through the reference viewpoint R, so that 
projection points existing on the same line are 
dealt with together as a projection point series. 
For example, as shown in Fig. 13, focusing attention 
20 on a line that passes through the reference 

viewpoint R, and assuming that a projection point of 
a projection plane Lj is and that a projection 
point series of the projection points is S, S can be 
represented as S={T j |jej). In addition, assuming 
25 that a set of projection point series is S, S£S 

holds true. 

After completing the processing of the 
step 503, next, the texture array keeping means 103c 
keeps, on the memory of the image generation 
30 apparatus, the array (texture array) for storing the 
images that are to be texture-mapped on the 
projection planes (step 503d). At this time, the 
array to be kept includes color information and 
existence probability information by 8 bits for each 
35 pixel as texture information corresponding to the 
position of the projection point. 

In the step S03d, correspondence between 
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the two-dimensional digital coordinates (U 3 ,V 3 ) of 
the pixel of the texture array and the three- 
dimensional coordinates (X^Y^Zj) of the projection 
point Ti is also set. At this time, for example, 
values of (Xj,*,,^) for every (U 3 ,V 3 ) may be set as 
a table. Or, only some values of (X 3 ,Y 3 ,Z 3 ) of 
representative (U 3 ,V 3 ) may be set and other 
correspondences are obtained by interpolation 
processing (linear interpolation, for example) . 

After completing the processing of the 
step 503d, color information and existence 
probability information for each pixel corresponding 
to each projection point kept in the step 503d are 
determined based on the images of the object 
obtained in the step 502. At that time, the 
projection point series S are scanned successively 
within a limit of S^E. Further, projection points 
Tj are scanned successively within a limit of T a €S, 
so that double loop processing is performed. 

In the loop processing, the projection 
point series S to be operated is initialized to a 
start position (step 503e) . Then, next, the 
projection point T 3 is initialized to a start 
position in the projection point series S, for 
25 example, initialized to j-1 (step 503f ) . 

After finishing the step 503e and the step 
503f, coordinates (X 3 W,Z 3 *) of the projection 
point T 3 is obtained, and each position on each 
image plane corresponding to the point at the 
position (X 3 \Y 3 *,Z 3 *>, when taking the point at the 
position <X 3 \Y 3 W> by each camera, is calculated 
using relationships from the equation 14 to the 
equation 17 (Step 503g) . At this time, a set of 
cameras for which the corresponding points are 
calculated is represented as 2 = {CiliSIJ. The set 
S of the cameras may be all cameras, or may be one 
or plural cameras selected from the cameras 
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according to positions of the virtual viewpoint P, 
the reference viewpoint R and projection points Tj . 

Assuming that corresponding points of each 
camera obtained here are Gij(i^I), and that the 
5 digital coordinates are represented by (Uij*,Vij*) (i^ 
I) . 

After completing the processing in step 
503g, next, the color information determination 
means 103e determines color information on a pixel 
10 (Uj*,Vj*) on the texture array corresponding to the 
projection point Tj by mixing color information in 
(uij^Vip*) (iSl) (step 503h) . The mixing processing 
is to calculate an average of pieces of color 
information of corresponding points of each camera, 
15 for example . 

In addition, in the mixing processing, it 
is possible to assign weights according to an angle 
9ij formed by the camera d, the projection point Tj 
and the reference viewpoint R . For example, as 
20 shown in Fig, 14, assuming that the set of camera is 
S - tCnrC n+ i,C n+ 2} (I*={n,n+l,n+2}) - In this case, 
assuming that vectors representing color information 
(R, G, B) at the projection point Tj and the 
corresponding point Gij are expressed by Kj and Kij 
25 respectively. When Kj is determined by the 

following equation 19, for example, the smaller the 
angle between the line from the reference viewpoint 
R to the projection point Tj and the line of sight 
of the camera is, the greater the degree 
30 contributing to the mixing processing is. 
[Equation 19] 

1 Yj C0S °V 

16/ 

After completing the processing in step 503h, the 
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corresponding point matching processing means 103d 
calculates the degree Q 3 of correlation of the 
corresponding points Syjiei) of each camera with 
respect to the projection point T 3 (step 503i) . 
When using the following equation 20, the degree Qj 
of correlation becomes a positive value, and the 
higher the correlation of the corresponding points, 
the smaller the Qj is. 
[Equation 20] 

in the equation 20, although color information of 
the projection point and color information of the 
corresponding point are compared at only one point, 
comparison of color information may be performed 
between plural points near the projection point and 
plural points near the corresponding point. At this 
time, for example, as shown in Fig. 14, when defining 
a region <K 3 near the projection plane T 3 and a 
corresponding region at the camera C*, degree Q 3 
20 of correlation of the regions can be calculated by 
the following equation 21, for example. 
[Equation 21] 

e = y y {Kiu^-K^v,)} 2 

wherein KtU^Vj) indicates an estimated value of 
color information at the coordinates (U,,Vi) in the 
texture array and K(u iU v ±i ) indicates color 
information at coordinates (u^Vj) in a photographed 

image by the camera d- 

The method for calculating the degree of 
correlation is not limited to the above-mentioned 
one, and the present invention is not dependent on a 
particular calculation method. For example, in the 
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example shown in Fig. 15, the region <&j is formed by 
8 pixels including the pixel of the projection point 
Tj and surrounding pixels, and the corresponding 
region is formed by 8 pixels including the pixel 
5 of the corresponding point and surrounding pixels. 
However, the defining method for the near region G>j 
and the corresponding region is not limited to 
this example. 

After completing the processing of the 

10 step 503i, the projection point Tj is updated (step 

503j), and it is determined whether every projection 
point T^S is scanned (step 503k) , If all points 
are scanned, the step proceeds to the step 5031- If 
all points are not scanned, the step returns to the 

15 step 503g. 

When it is determined that all points have 
been scanned in step 503k, next, the existence 
probability information determination means 103f 
determines, based on the degree Qj of correlation 

20 calculated in step 503i, degree (existence 

probability information) (Jj of probability of 
existence of the object on the projection point with 
respect to every projection point Tj(jEJ) on the 
line that passes through the reference viewpoint R 

25 (step 5031) . In the determination , the existence 
probability information pj needs to satisfy the 
conditions of the equation 22 and the equation 23. 
[Equation 22] 

0 < /? y < 1 

30 [Equation 23] 
M 

7=1 



Further, the higher the probability of existence of 
the object at the projection point Tj is, the nearer 
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to 1 the probability is. Therefore, the existence 
probability information Mj^J) is obtained by 
performing conversion processing represented by the 
following equation 24 and the equation 25 on the 
degree Q a of correlation between the projection 
point and the corresponding points calculated in the 
step 503i, for example. 
[Equation 24] 

>,- 1 



10 [Equation 25] 

* =-^_ 

Pj M ^ 

However, since the existence probability information 
P 3 only needs to satisfy the conditions of the 
equation 22 and the equation 23, the conversion 
15 processing is not necessarily limited to the method 
represented by the equation 24 and the equation 25. 

After completing the processing of the 
step 5031, next, the projection point series S is 
updated (step 503m), and it is determined whether 
all projection point series ses are scanned (step 
503n) . If all projection point series are scanned, 
the step proceeds to the next step 503o. If all 
projection point series are not scanned, the step 
returns back to step 503f. 

If it is determined that all projection 
point series are scanned in step 503n, next, the 
rendering means 103g renders and generates an image 
obtained by viewing the projection planes 
L 3 (j=l,2,...,M) having multi-layered structure from 
30 the virtual viewpoint P according to the existence 
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probability information p 3 (step 503o) , For example, 
as shown in Fig. 16, assuming that coordinates of the 
virtual viewpoint P on the image are represented as 
<Up,v p ) . At this time, color information K p * of a 
5 pixel p*(u p *,v p *) on the image is determined by 
multiplying color information {Kj*lj^J} of 
projection point series {Tj'lj^J} on the line 
connecting p and p* with existence probability 
information {pj*|j^J} and by adding the multiplied 
10 color information, so that the color information K p * 
is represented by the following equation 26. 
[Equation 26) 

By determining color information for every pixel on 
the image plane, the image for the virtual viewpoint 
P is obtained. 

In addition, instead of using the equation 

26, by calculating K p * using the following equation 

27, it is guaranteed that K p * is within an effective 
color space even when the reference viewpoint R and 
the virtual viewpoint P are different. 

[Equation 27] 

M 

K P =J ~M 

IX 

Although a procedure in which color information is 
25 determined by scanning the pixels on the image plane 
is shown, it is not limited to the procedure. For 
example, rendering processing can be relied on 
general graphic library such as OpenGL, DirectX and 
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the like by passing data such as configuration of 
projection planes, texture array and setting of 
viewpoint P and the like to the graphic library. 

Then, generation processing (step 503) of 
5 the virtual viewpoint image ends, so that the 

generated virtual viewpoint image is displayed on 
the image display means 4 (step 504). As shown in 
Fig. 17, for example, in the virtual viewpoint image 
6 displayed on the image display means 4, a portion 
10 7A in which the degree Qj of the correlation 

calculated in step 5031 is low, that is, in which 
reliability of estimation value is low, is rendered 
obscurely and appears blurred in the image 7 of the 
object. Therefore, for example, a part does not 
15 appear to lack like the conventional virtual 
viewpoint image 6 as shown in Fig.l, and 
deterioration is not annoying in the eyes of the 

user. 

After that, it is determined that process 
20 continues or stops in step 505. If the process is 
continued, the process is repeated from the first 
step 501. If the process should be ends, the 

process ends. 

As mentioned above, the virtual viewpoint 
25 image generation method using the virtual viewpoint 
image generation apparatus of this embodiment 1-1 
does not intend to obtain the accurate geometrical 
model of the object for every case and for every 
portion like the conventional measure. In contrast, 
30 the virtual viewpoint image generation method 
assumes that estimation values having enough 
reliability cannot be obtained in distance 
estimation according to photographing conditions and 
portions of the object. Thus, a portion in which an 
35 estimation value having low reliability is obtained 
is rendered vaguely so that the portion contributes 
little to image generation to prevent extreme image 
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deterioration. On the other hand, a portion in 
which distance data having high reliability is 
obtained is rendered clearly so that the portion 
highly contributes image generation. Therefore, 
5 deterioration of image of the portion having low 

estimation reliability can be inconspicuous so that 
the virtual viewpoint image having small 
deterioration can be obtained in the eyes of the 
user - 

10 In addition, the virtual viewpoint image 

generation apparatus 1 of this embodiment 1-1, since 
the virtual viewpoint image can be generated using 
texture mapping, workload of the apparatus in the 
image generation processing can be decreased and the 

15 virtual viewpoint image can be generated at high 
speed . 

In addition, the virtual viewpoint image 
generation apparatus 1 does not need to be a 
specific apparatus. For example, the virtual 

20 viewpoint image generation apparatus 1 can be 

realized by a computer including a CPU and a storing 
device such as a memory and a hard disk and the like, 
and a program, for example. In such a case, by 
generating a program that can cause the computer to 

25 execute each step shown in Fig. 11 and by executing 
the program on the computer, the virtual viewpoint 
image having small image deterioration can be easily 
generated at high speed even though the computer is 
a general personal computer. In this case, data for 

30 the processing is held in the storing apparatus and 
read as necessary so as to be processed by the CPU. 

The program can be provided by storing in 
a recording medium such as a floppy disk or CD-ROM, 
and can be provided via a network. 

35 The configuration of the virtual viewpoint 

image generation apparatus, the generation method 
and the processing procedure of the virtual 
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viewpoint image described in this embodiment 1-1 are 
merely examples. The principal object of the 
present invention is to determine transparency 
information of the multi-layered projection planes 
5 according to reliability of corresponding regions 
among images obtained by taking the object from 
plural different viewpoints. Thus, within a limit 
that does not largely differ from the scope, the 
present invention is not dependent on a particular 
10 processing method or implementation. 

In addition, systems using the virtual 
viewpoint image generation apparatus 1 are not 
limited to the system of one direction shown in 
Fig. 7. The present invention can be applied to a 
15 bi-directional system. 

Fig. 18 is a schematic diagram showing an 
application example of a system to which the virtual 
viewpoint image generation apparatus 1 of this 
embodiment 1-1 is applied. 
2Q The virtual viewpoint image generation 

apparatus 1 of the embodiment 1-1 is applicable for 
a system, such as TV phone, TV conference and the 
like, for example. For example, as shown in Fig. 18, 
the virtual viewpoint image generation apparatus 1 
25 can be applied to a system for supporting visual 
communications in which a user A and a user B 
remotely existing via a communication network 
present an image with each other while both users 
regard the other party as a user and an object. In 
30 the system, assuming that an image of the user B 
viewed from a viewpoint desired by the user A is 
Img[A-*B], lmg[A-B] is generated based on an image of 
the user B photographed by the object taking means 
(camera) 3B in the user B side, so that the image is 
35 provided to the image display means 4A in the side 

of the user A. Assuming that an image of the user A 
viewed from a viewpoint desired by the user B is 
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Img[B-A], Img [B-A] is generated based on an image of 
the user A photographed by the object taking means 
(camera) 3A in the user A side, so that the image is 
provided to the image display means 4B in the side 

5 of the user B. 

In the system shown in Fig. 18 , an example 
is shown in which the viewpoint position input means 
of each User is configured by a data transmission 
means 201A, 201B and a data receiving means 202A, 
10 202B in a position/posture sensor mounted on a head 
of the user, and the viewpoint position input means 
automatically follows movement of the head of the 
user so as to calculate a desired virtual viewpoint. 
However, the viewpoint position input means does not 

15 necessarily adopt such configuration- The same 

function can be realized by estimating the position 
/ posture of the head based on the image of the user 
photographed by the object taking means 3A, 3B. 

In the system, Img[A->B] can be generated 

20 in any of the virtual viewpoint image generation 
apparatus 1A in the user A side and the virtual 
viewpoint image generation apparatus IB in the user 
B side. In the former case, the image of the user B 
taken by the camera 3B is transmitted to the virtual 

25 viewpoint image generation apparatus 1A in the user 
A side via the network 8, and the Img [A-*B] is 
generated by the virtual viewpoint image generation 
apparatus 1A based on the image and is presented by 
the image display means 4A. In the latter case, the 

30 image of the user B taken by the camera 3B in the 
user B side is generated by the virtual viewpoint 
image generation apparatus IB in the user B side- 
After that, the virtual viewpoint image Img[A->B] is 
transmitted to the virtual viewpoint image 

35 generation apparatus 1A in the user A side and is 
presented by the image display means 4A- In 
addition, although not described in detail, similar 
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procedure can be adopted for Img[B-»A]. 

In addition, each means forming the image 
generation means 103 in Fig. 6 can be provided in any 
one of the virtual viewpoint image generation 
5 apparatus 1A in the user A side and the virtual 

viewpoint image generation apparatus in the user B 
side. For example, for generating the Img[A->B], the 
projection plane determination means 103a, the 
reference viewpoint determination means 103b and the 

10 corresponding point matching means 103d are provided 
in the image generation apparatus 1A in the user A 
side, and the texture array keeping means 103c, the 
color information determination means 103e, the 
existence probability information determination 

15 means 103f and the rendering means 103g can be 

provided in the image generation apparatus IB in the 
user B side. Although detailed description is not 
provided, similar configuration can be adopted for 
Img [ B->A] . 

20 In addition, in any place on the network 8, 

an image generation apparatus 1C can be provided 
separately from the virtual viewpoint image 
generation apparatuses 1A and IB, so that all or a 
part of image generation means can be provided in 

25 the image generation apparatus 1C. 

Although communications between two users 
of user A and user B are described, the number of 
users are not limited to two. The present invention 
can be applied to more users. In the case, assuming 

30 a" virtual space used for communication separately 

from the actual space in which users actually exist, 
by providing an image of other user according to 
position relationship with each other, the system 
can provide users with feeling as if to share 

35 virtual space (cyber space) on the network. 
(Embodiment 1-2) 

Figs.l9A and 19B are schematic diagrams 



20 05$12JH6B 2iet42ft 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 74 



-64- 



10 



for explaining a virtual viewpoint image generation 
method of the embodiment 1-2. Fig.l9A is a flow 
diagram indicating processing that is a feature of 
this embodiment 1-2. Fig.l9B is a flow diagram 
showing an example of concrete processing procedure 
of steps for determining transparency information. 

In the embodiment 1-2, an example is 
described in which image generation is performed by 
converting the existence probability information 
into transparency information instead of using the 
existence probability information of the projection 
point calculated in the step 5031 in generation 
processing for virtual viewpoint image described in 
the embodiment 1-1. 
15 In this example, as to the configuration 

of the virtual viewpoint image generation apparatus 
1 and the general processing procedure, forms 
similar to examples described in the embodiment 1-1 
can be adopted. Thus, only parts having differences 
20 with respect to the embodiment 1-1 are described in 
the following. 

In the embodiment 1-1, in step 503 for 
generating the image, the virtual viewpoint image is 
generated using the existence probability 
25 information Pj determined in step 5031 as shown in 
Fig. 11. On the other hand, in this embodiment 1-2, 
as shown in Fig.l9A, a step 503p is added for 
determining transparency by converting the existence 
probability information after the step 5031. 
30 Therefore, although the array for storing 

color information and the existence probability 
information is kept in the step 503d for keeping 
texture array in the embodiment 1-1, the array is 
kept for storing color information and the 
35 transparency information in the step 503d in this 
embodiment 1-2. 

The transparency information aj is 
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calculated based on the existence probability 
information 0j . In the same way as the step 5011 of 
the embodiment 1-1, also in this embodiment 1-2, the 
existence probability information is calculated once 
5 in the step 5031 and the transparency information is 
calculated in the next step 503p> 

In step 503c for performing rendering 
processing in this embodiment 1-2, Dj is 
successively calculated according to equations from 

10 the equation 2 to the equation 4 instead of using 

the equation 26 and the equation 27 described in the 
embodiment 1-1. Therefore, color information Kj* of 
the pixel p # (u p ",v p *) on the image plane is 
calculated using the following equation 28. 

15 [Equation 28] 

= a M K M + {l-a M )a M _ x K M _ x + — 

+ (l-a M X 1 - a Af-l)-( 1 -«2)^l 

The image generation method in this 
embodiment is described so far. An example of a 
method for calculating the transparency information 
20 «j based on the existence probability information (Jj 
is described as follows . 

First, by comparing the equation 26 and 
the equation 28, the following equation 29 is 
obtained - 
25 [Equation 29] 



M 



a 



m 



) 



•a s (jej) 



m- 

Based on this relationship, a procedure for 
obtaining cxj in an order of j^M, M-l, 1 is as 
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f ollows . 

First, as shown in Fig.l9B, M is assigned 
to j (j=M) as an initial value of j (step 5031p) . 
Next, a K is determined as 0£«=(3 M from the equation 29 
5 (step 5032p) . Next, the value of j is updated to 
j=j-l (step 5033p) . 

Next, it is determined whether cx j+ i is 1 
(step 5034p) . At this time, if ccj+i^l, a 3 is 
determined by the following equation 30 from the 
10 relationship of the equation 29 (step 5035p) . 
[Equation 30] 

a i =—*r^ Pi 

no-"") 

On the other hand, if a j+ i=l, aj is determined by the 
following equation 31 (step 5036p) 
15 [Equation 31] 

a j =1 

The reason is described as follows. First, when 
a j+ i=l, the following equation holds true. Thus, the 
equation 30 cannot be calculated since the 
20 denominator is 0, 
[Equation 32] 

TJ(i-aJ=o 

Thus, the following equation 33 is obtained by 
^ expanding the equation 32, Then, the equation 34 is 
25 obtained by substituting the equation 29. 
[Equation 33] 
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[Equation 341 

The following equation 35 is obtained from the 
equation 34, the equation 22 and the equation 23. 
5 [Equation 35] 

By substituting the equation 32 and the equation 35 
into the lower side of the equation 29, 0=0*aj is 
obtained, so that it can be understood that a 3 can 
10 take any value* Therefore, in this embodiment 1-2, 
<Xj is set as 1 («j = l) , for example. 

However, as mentioned above, aj can be set 
as any value, and the present invention is not 
dependent on a particular determining method of Ofj. 
15 Next, it is determined whether processing 

completes up to j-1 (step 5037p) . When every 
processing completes, the procedure ends, when every 
processing does not end yet, the procedure returns 
to the step 5033p- 
20 As described above, according to the 

virtual viewpoint image generation method of the 
embodiment 1-2, like the embodiment 1-1, the virtual 
viewpoint image in which partial image deterioration 
is inconspicuous can be generate easily and at high 
25 speed. 

As described in the embodiment 1-1, in 
image generation using the existence probability 
information as it is, there may be a case in which 
brightness increases near the blocking area of the 

30 object when the reference viewpoint and the virtual 
viewpoint are different. On the other hand, as 
described in the embodiment 1-2, in the image 
generation for converting the existence probability 
information to transparency, there is an effect to 

35 prevent the phenomenon. Therefore, a virtual 
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viewpoint image having small image deterioration and 
more similar to the actual object can be obtained. 

In addition, as described in the 
embodiment 1-1 , in the image generation using the 
5 existence probability information as it is, there is 
no guarantee that color information calculated by an 
equation described later falls within an effective 
limit when the reference viewpoint and the virtual 
viewpoint are different, so that correction 
10 processing is required, for example. On the other 
hand, according to the image generation for 
converting the existence probability information to 
transparency like the embodiment 1-2 , such 
correction is not necessary. Therefore, the image 
15 generation processing can be simplified. 

In addition, in image generation for 
converting the existence probability information to 
the transparency like the virtual viewpoint image 
generation method of this embodiment 1-2, a 
20 semitransparent object can be efficiently 

represented, so that there is an effect that the 
present invention can be applied to more objects in 
the actual world. 

The virtual viewpoint image generation 
25 method described in this embodiment 1-2 is an 

example, and the principal object of the present 
embodiment is to generate the virtual viewpoint 
image by converting the existence probability 
information to the transparency. Therefore, the 
30 present invention is not dependent on particular 

calculation method or processing procedure within a 
limit largely exceeding the object. 

The color information corresponds to 
brightness information for black and white images, 
35 and the same procedure can be applied. 
(Embodiment 1-3) 

Fig. 20 is a schematic diagram for 
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explaining the virtual viewpoint image generation 
method of the embodiment 1-3. The figure shows 
examples of projection plane group, reference 
viewpoint, virtual viewpoint and projection points. 

In the embodiment 1-3, a method for image 
generation is described in which the projection 
plane Lj common for plural cameras is not used but 
projection planes specific for each camera Cj are 
set. Since the configuration of the virtual 
viewpoint image generation apparatus 1 and general 
processing procedure for image generation in this 
embodiment are the same as those described in the 
embodiment 1-1, detailed descriptions are not 
provided . 

15 First, as described in the embodiment 1-1, 

the virtual viewpoint is determined in the step 501, 
and the images of the object is obtained in the next 
step 502. 

In the image generation method of this 
20 embodiment 1-3, projection plane groups specific for 
each camera are set in the next step 503 in 
generation of the virtual viewpoint image. 

At this time, as shown in Fig. 20, for 
example, the projection plane groups are set as 
25 projection plane groups Ai={Li 3 1 j e j} each parallel 
to the image surface Imgi(iei) specific to each 
camera d I={n~l, n,n+l, n+2} ) . 

After setting the projection plane groups, 
next, in processing for determining the reference 
30 viewpoint in the step 503b, each reference viewpoint 
Ri specific to the projection plane group At is set 
in a position the same as the viewpoint C ± of the 
camera . 

Next, after completing the step 503b, 
35 processing of the 503c is performed according to the 
procedure described in the embodiment 1-1. Then, in 
the next step 503d, each pixel of the digital image 
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taken by the camera is reverse-projected on the 
projection planes so that the pixel of the digital 
image is associated with each pixel of the texture 
array of the projection planes. 
5 Conversion from the point (u,v) of the 

digital image to the point (x,y> on the image plane 
is represented by the equation 17, for example. 
Reverse-projection from (x,y) to the point (X,Y,Z) 
on the projection plane in the three-dimensional 
10 space can be mathematically expressed as follows, 

for example. 

Generally, when a two-dimensional point 
(x,y) is given, there are an infinite number of 
three-dimensional points that satisfy the equation 
15 14 and the equation 15. In the points, points 

(X,Y,Z) existing on the projection plane are the 
reverse-projection image. 

An equation of the projection plane is 
generally expressed as aX+bY+cZ+d=0 . A following 
20 equation 36 is obtained by rewriting this equation 
by vector representation. 
[Equation 36] 



25 



[abed] 



X 
Y 
Z 
1 



= 0 



By integrating the equation 14, the equation 15 and 
the equation 36, the following equation 37 can be 
obtained . 

[Equation 37] 
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The reverse-projection image from (x,y) to (X,Y,Z) 
can be obtained by solving the equation 37 with 
respect to (X,Y,Z). When the matrix of 4 rows and 4 
5 columns in the equation 37 has an inverse matrix, by 
substituting S' for 1/s, the reverse-projection 
image can be obtained by the following equation 38- 
[Equation 38] 
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10 The above-mentioned example is an example to the 

utmost. Calibration for correcting aberration of a 
lens (distortion/ for example) can be performed, and 
the point (X,Y,Z) on the projection plane 
corresponding to the point (u,v) of the digital 

15 image can be held as a table . 

After that, processing from the step 503e 
to the step 503g is performed according to the 
procedure described in the embodiment 1-1- Then, in 
the 'processing for determining the color information 

20 in the next step 503*1, color information of the 

projection points on the projection plane group A ± 
are determined only by using the color information 
of the image taken by the camera C±. 

By performing the procedures of the step 

25 503d and the step 503h described in this embodiment 
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1-3, the digital image taken by the camera can be 
used as the color information of the texture array 
of the projection plane as it is. 

After that, again, the procedure from the 
5 step 503i to the" step 503h is performed in the same 
way as the embodiment 1-1. Then, next, in the 
rendering means in the step 503o, mixing processing 
for color information is performed with respect to 
projection points overlapping when viewed from the 

10 virtual viewpoint P. At this time, for example, in 
the example shown in Fig. 20, mixing processing for 
color information is performed on a line. passing 
through the virtual viewpoint P for projection 
points on* the projection plane groups A n and A n +i- 

15 Assuming that a projection point in the 

projection plane Lij is T ij# color information at T±j 
is Kij, and that existence probability information 
is [Sij, color information of the image plane of the 
virtual viewpoint P represented as the equation 27 

20 in the embodiment 1-1 is determined in the following 
way, for example. 

That is, the color information K p * of a 
pixel p*(u p * / Vp*) on the image plane is determined as 
an- weighted average value of color information 

25 calculated by using weight coefficients of existence 
probability information { I i^I, j ^ J} 
corresponding to color information {Kij* I i^I, j ^ J} 
of projection point series {T ±j * | i^I, j e J} on a line 
connecting between P and p* as shown in the 

30 following equation 39. 
[Equation 39] 
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As described above, according to the 
virtual viewpoint image generation method of this 
embodiment 1-3 , the virtual viewpoint image in which 
5 partial image deterioration is inconspicuous can be 
generated easily and at high speed like the 
embodiment 1-1. 

In addition, by setting the projection 
plane groups specific to each camera irrespective of 
10 position relationship among cameras like this 

embodiment 1-3, even if the placement of the cameras 
are complicated and irregular, processing of setting 
the projection plane groups is not affected so that 
image generation can be performed by the consistent 
15 processing method. 

The virtual viewpoint image generation 
method described in this embodiment 1-3 is an 
example, and the principal object of the present 
embodiment is to generate the virtual viewpoint 
20 image by converting the existence probability 

information to the transparency. Therefore, the 
present invention is not dependent on particular 
calculation method or processing procedure within a 
limit largely exceeds the object. 
25 The color information corresponds to 

brightness information in the case using black and 
white images, and same procedure can be applied. 
(Effects of first embodiment) 
The method described in the first 
30 embodiment does not intend to obtain a geometrical 
object model accurate for every case and every 
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portions like the conventional method. But, under 
the assumption that an estimated value having 
adequate reliability cannot be obtained in the 
distance estimation depending on picture-taking 
5 conditions and portions, a portion by which an 

estimated value of low reliability is obtained is 
drawn vaguely so as to provide the portion with low 
contribution to image generation for avoiding 
extreme image deterioration, and a portion, by which 
10 distance data of high reliability is obtained, is 

drawn clearly so as to provide the portion with high 
contribution to image generation. Therefore, image 
deterioration of the poison of low estimation 
reliability can be inconspicuous, 
15 m addition, the problem that the 

brightness increases near the blocking area when the 
reference viewpoint and the virtual viewpoint are 
different can be solved. When the reference 
viewpoint and the virtual viewpoint are different, 
20 there is no guarantee that calculated color 

information falls within an effective limit of color 
information. Thus, correction processing may be 
necessary. On the other hand, according to the 
method described in this embodiment, such correction 
25 is unnecessary- In addition, a semitransparent 

object can be efficiently represented, so that there 
is an effect that the present invention can be 
applied more widely to more objects in the actual 
world. In addition, since the groups of the 
30 projection planes specific to each camera are set 

irrespective of position relationship among cameras, 
even if placement of the cameras is complicated or 
at random, processing for setting the groups of the 
projection planes is not affected so that image 
35 generation can be performed by consistent processing 
method - 

Further, when setting the projection plane 
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groups specific to each camera, it is not necessary 
to perform mixing processing among images taken by 
corresponding cameras as to color information of the 
projection planes. Therefore, parallel processing 
5 can be performed when performing processing by a 

computer, so that speed of the image generation can 
be increased. 

In addition, since color information of a 
projection plane group associated with the same 
10 camera are the same, the texture memory for storing 
the color information can be shared. Therefore, the 
memory is not so consumed even when the number of 
projection planes increases, so that workload of the 
apparatus used for image generation can be decreased. 
15 In addition, since a camera corresponding to a 

projection plane is uniquely determined, calibration 
such as correction of distortion of a lens can be 
performed easily and quickly by setting 
correspondence of coordinates of them beforehand. 
20 In addition, processing time for 

performing the processing by the apparatus for 
generating the virtual viewpoint image based on the 
images of the object can be decreased, or the 
workload of the apparatus can be decreased, so that 
25 even a general personal computer can generate an 

image having small partial deterioration in a short 
time - 

[Second embodiment] 

In the following, the second embodiment of 
30 the present invention is described. The second 

embodiment mainly corresponds to the claims 12-21. 
Although the basic mechanism of the second 
embodiment is the same as that of the first 
embodiment, the second embodiment is characterized 
35 in that plural groups of camera sets are provided 

and the existence probability is calculated based on 
correlation degree obtained for each camera set. In 
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figures for explaining the second embodiment, the 
same reference signs are assigned for the same 
functions - 

The image generation method of the second 
5 embodiment is a method for obtaining, from plural 
images of different viewpoints, a three-dimensional 
shape of an object appearing in the images, and 
generating an image for providing the threer 
dimensional image of the object or generating an 

10 image of the object viewed from arbitrary viewpoints - 
The three-dimensional shape of the object is 
obtained by estimating distance from the observer's 
viewpoint to each point of the surface of the object 
by setting multi-layered projection planes using the 

15 texture mapping method- When estimating the 

distance to the surface of the object, correlation 
degree between the projection point and each 
corresponding point of the image (to be referred as 
corresponding point hereinafter) is obtained for 

20 each of points (to be referred to as projection 
point hereinafter), on the projection planes, 
overlapping when viewed from the observer' s 
viewpoint- Then, an projection point near which the 
surface of the object exists is estimated among the 

25 overlapping projection points according to the 
correlation degree of the projection points 
overlapping when viewed from the observer's 
viewpoint. In the image generation method of this 
embodiment, it is not considered that the surface of 

30 the object exists at a neighborhood of projection 
point among plural projection points overlapping 
when viewed from the observer's viewpoint- But, it 
is considered that the surface of the object exists 
in a neighborhood of each projection point in a 

35 ratio according to the correlation degree of each 

projection point. In the image generation method of 
this invention, a probability (to be referred to as 
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existence probability) that the surface of the 
object exists at the projection point or the 
neighborhood is determined according to the 
correlation degree of each projection point. Then, 
5 when the image is generated based on the three- 
dimensional shape of the object, the color 
information of the projection point is assigned to 
the color information of each point of the image to 
be generated according to the value of the existence 

10 probability. Accordingly, from the viewpoint of the. 
observer who observes the projection planes, a 
portion at which estimation of the distance to the 
object surface is difficult is rendered vaguely, so 
that discontinuous noise and the like becomes 

15 inconspicuous. 

In addition, with respect to the plural 
projection points overlapping when viewed from the 
observer's viewpoint or from the reference viewpoint 
for obtaining the three-dimensional shape of the 

20 object, if probability density distribution of the 
existence probability of the object surface can be 
estimated to some extent, the existence probability 
can be obtained using a parameter function p(l) to 
which the probability density distribution is 

25 reflected- In such a case, variation of the 

correlation degree due to noise of the taken images 
can be decreased so that deterioration of 
reliability of the existence probability can be 
prevented. 

30 In addition, instead of obtaining the 

correlation degree for a projection point using 
corresponding points of all images, by obtaining it 
using corresponding points of images taken from some 
predetermined viewpoints, reliability of the 

35 correlation degree improves so that reliability of 
the existence probability improves by removing each 
image in which the object does not appear due to 
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occlusion even though the surface of the object 
exists on the projection point. 

Figs 21-28 are schematic diagrams for 
explaining the principle of the image display method 
in this embodiment. Fig. 21 is a diagram for 
explaining a concept of a method for generating the 
image to be displayed. Fig. 22 is a diagram 
representing Fig. 21 in two-dimensional. Fig.23A and 
Fig.23B are diagrams for explaining a method for 
obtaining the correlation degree of the 
corresponding point- Fig.24A and Fig.24B are 
diagrams for explaining a problem when obtaining the 
correlation degree of the corresponding point. 
Fig. 25 is a diagram for explaining a method for 
15 solving the problem when obtaining the correlation 
degree. Fig.26A and Fig.26B are diagrams for 
explaining an example of a method for ^improving 
accuracy of the existence probability. Fig.27A, 
Fig-27B and Fig.2B are diagrams for explaining 
characteristics of this embodiment. 

In the image display method of this 
embodiment, when generating the image to be 
displayed, first, a virtual three-dimensional space 
is set in the image generation apparatus such as a 
computer. Then, on the three-dimensional space, 
viewpoints d (i-l r 2, ...,N) of cameras that take the 
images of the object and projection planes 
L j (j=l,2,...,M) having a multi-layered structure for 
estimating the three-dimensional shape of the object 
are set. At this time, when the cameras are 
arranged on a line, the viewpoints C are set on the 
X axis (z=0) as shown in Figs. 21 and 22, for example. 
As the projection planes lj, planes parallel to the 
XY plane Z=l 3 (<0) are set as shown in Figs. 21 and 22, 

35 for example. 

As shown in Figs. 21 and 22, if there is 
the surface of the object at an intersection point 
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T m between a line Ip extending from the observer's 
viewpoint p and a projection plane L ro/ the point 
should be taken on a point (corresponding point) G ifin 
on the image photographed by the camera placed at 
5 the viewpoint Ci« In the same way, when the surface 
of the object exists at the projection point T m , the 
point should appear at each of corresponding points 
G i+ i, m and G i+2/ m on images taken by the cameras placed 
at the viewpoints C i+ i and C i+2 respectively, 

10 Therefore, if the correlation degrees (similarity) 
of the corresponding points Gi, m , G i+ i flB and Gi+ 2f m on 
each image for the projection point T m are obtained, 
it can be estimated whether the surface of the 
object exists at the projection point T n or the 

15 neighborhood- By performing the estimation for each 
projection point Tj on the line lp extending from 
the observer's viewpoint P, a neighborhood of a 
projection point Tj (projection plane Lj) at which 
the surface of the object exists can be estimated on 

20 the line lp extending from the observer's viewpoint 
P- 

When estimating a neighborhood of a 
projection point Tj (projection plane Lj) , on the 
line lp extending from the observer's viewpoint p r 

25 where the surface of the object exists, correlation 
degree Qj of each corresponding point G if j 
corresponding to the projection point Tj is used. 
The correlation degree Qj is obtained using the 
following equation 40 in the same way as the first 

3 0 embodiment « 

[Equation 40] 

i*I 

wherein I is a set of viewpoints Ci that can define 
corresponding points G±,j for the projection point Tj 
35 on the image planes in the viewpoints C ± (i=l, 2,...,N) , 
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K i:i is color information of each corresponding point 
Gi, jf and Kj is color information of the projection 
point Tj and is an average value of color 
information K tj of each corresponding point G A/j . 
5 At this time r a case is considered in 

which the result shown in Fig.23A is obtained, for 
example, when superimposing the actual surface shape 
of the object on the established projection plane L 3 . 
At this time, in projection points Tj on the line lp 
10 extending from the observer's viewpoint p, the 
projection point T m is the nearest to the object 
surface that can be viewed from the viewpoints Ci, 
C i+ i and C i+2 of the cameras* Thus, as shown in 
Fig.23A, points of the object surface appearing in 
15 the corresponding points Gj., m r Gi+i, m anci G i+2,m 

corresponding to the projection point T m are very 
close with each other. On the other hand, for 
example, the points of the object surface appearing 
in corresponding points corresponding to the 
20 projection point T 2 are apart with each other. As a 
result, when obtaining the correlation degree Qj of 
each projection point Tj on the line lp, only the 
correlation degree Q a for the projection point T* is 
very small value as shown in Fig.23B. Therefore, 
25 when being viewed from the observer' s viewpoint P in 
the direction of the line lp, the surface of the 
object can be estimated to exist at a position of 
the projection point T m , that is, at a a distance l m 
at which the projection plane L ffi is set. 
30 Thus, by providing lines lp from the 

observer's viewpoint P in various directions and 
repeating similar processing for projection points 
Tj on each line lp, the surface shape of the object 
appearing in the images can be estimated. 
35 However, such estimation method is 

effective, that is, the reliability of the estimated 
surface shape of the object is high, only in the 
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case where the actual surface shape of the object is 
relatively simple as shown in Fig.23A and 23B. When 
the, shape of the object is complicated or when 
plural objects overlap when being viewed from the 
5 observer's viewpoint, the reliability of the 

estimated surface shape of the object is lowered. 

As an example in which the reliability of 
the estimated surface shape is lowered, a case is 
considered in which the result shown in Fig.24A is 
10 obtained, for example, when two objects overlaps 
with respect to the observer's viewpoint and when 
superimposing the projection planes L 5 with the 
surface shapes of the two objects. In this case, it 
can be considered that the correlation degree Q 3 for 
15 the projection points Tj on the line lp extending 
from the observer's viewpoint P as indicated by a 
dotted line in Fig.24A provides distribution shown 
in Fig-23B, for example. Thus, it can be considered 
that the reliability of the estimated surface shape 
20 of the object A is high near the line lp. 

However, when obtaining the correlation 
degree Q' m for the projection point T' to on the line 
lp' extending from the -observer' s viewpoint P as 
shown by the solid line shown in Fig.24A, the 
25 surface of the object B appears at the corresponding 
point G'i^of the image taken from the viewpoint d 
and the surface of the object A appears at the 
corresponding points G'i+i,* and G'i^.mOf the images 
taken from the viewpoints Ci+i tJa and Ci+2,m- In such a 
30 case, the correlation degree Q T m obtained by the 
equation 40 becomes large. As a result, 
distribution of the correlation degree Q'j of the 
projection points T ' -j on the line lp' becomes one 
shown in Fig.24B, so that it is difficult to 
35 estimate a neighborhood of a projection point T-j at 
which the surface of the object exists. At this 
time r when it is estimated that the surface of the 



2005412E16B 21B|47# ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 92 



-82- 

object exists near a projection point T'j having the 
smallest correlation degree Q'j like the case having 
the distribution shown in Fig.23B, discontinuous 
noise appear on the image to be displayed if the 
5 estimation is not correct. 

Thus, according to the image display 
method in this embodiment, such estimation is not 
performed in which the- surface of the object is 
estimated to exist at the projection point Tj or the 
10 neighborhood at which the correlation degree Qj is 

the smallest, but it is considered that the surface 
of the object exists at each projection point Tj in 
a probability according to the ratio of the size of 
the correlation degree Qj - At this time, assuming 
15 that the probability (existence probability) that 
the surface of the object exists at the projection 
point Tj or the neighborhood is (Jj, the existence 
probability pj of projection points on the line lp 
extending from the observer's viewpoint P, that is, 
20 projection points Tj overlapping when viewed from 
the observer's viewpoint P, requires to satisfy 
conditions as shown in the following equations 41 
and 42. 
[Equation 41] 

0</?,£l 

[Equation 42] 
M 

For the existence probability p 3 to take a value 
nearer to 1 as the probability of existence of the 
30 object surface is higher in projection points Tj, 
the existence probability 0 3 of each projection 
point Tj is determined by performing conversion 
processing represented by the following equations 43 
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and 44 for the correlation degree Qj of each 
projection point Tj . 
[Equation 43] 

1 

A = — 

1 Qj 

[Equation 44] 
"j M 



M 

The existence probability pj only needs to satisfy 
the conditions of the equations 41 and 42. Thus, 
the existence probability Pj can be determined in a 

10 method other than the conversion processing 
represented by the equations 43 and 44. 

According to such processing, by 
determining the probability Pj that the surface of 
the object exists at the projection point Tj or the 

15 neighborhood, the color information Kj and the 

existence probability Pj for each projection point Tj 
on the line lp extending from the observer's 
viewpoint P can be determined. 

When displaying a three-dimensional image 

20 of an object on a DFD of brightness modulation type 
by using the three-dimensional shape of the object 
estimated in this way, for example, each pixel 
corresponding to each projection point Tj in plural 
display planes is displayed with brightness 

25 according to the color information Kj and the 
existence probability K j . Accordingly, as to 
correlation degrees Qj of each projection point Tj on 
the line lp, when only a correlation degree Q m of a 
projection point T m is far different from other 

30 correlation degrees as shown in Fig.23B, only the 
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existence probability 3m of the projection point T m 
takes a large value. Therefore, brightness of only 
a pixel corresponding to the projection point T a 
becomes large so that the pixel looks clear for the 
5 observer who observes the projection plane L-, from 
the observer's viewpoint. 

On the other hand, when correlation 
degrees Qj of each projection point Tj on the line lp 
are as shown in Fig.24B so that it is difficult to 
10 estimate a projection point Tj near which the 

surface of the object exists, plural projection 
points having similar existence probability appear. 
Therefore, plural pixels corresponding to the 
projection points Tj on the projection planes Lj are 
15 displayed with similar brightness, so that sense of 
distance of the pixels becomes vague when viewed 
from the observer who observes the projection planes 
L 3 from the observer's viewpoint. However, since 
the surface image of the object is displayed on 
20 plural projection points overlapping when viewed 

from the observer's viewpoint P, discontinuous noise 
that may appear due to an error of estimation for 
the distance of the object surface does not appears. 
Therefore, even though the accurate three- 
25 dimensional shape of the object to be displayed is 
not obtained, a three-dimensional image of the 
object that looks natural for the observer can be 
displayed. 

In addition, when displaying the two- 
30 dimensional image (arbitrary viewpoint image) of the 
object viewed from an arbitrary viewpoint using the 
three-dimensional shape of the object estimated by 
the procedure, color information of a cross point 
between the line lp and the image plane of the image 
35 to be displayed is determined as color information 
obtained by mixing color information Kj of each 
projection point Tj on the line lp extending from 
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the observer' s viewpoint in a ratio of the existence 

probability (ij. 

If the probability density distribution of 
the probability that the surface of the object 
exists can be estimated to some extent when 
determining the existence probability (ij, error of 
estimation due to noise of the images taken at the 
viewpoints Ci can be decreased by performing 
statistical processing, based on the shape 
distribution of the object, on the existence 
probability of each projection point Tj determined 
by the equations 43 and 44. To differentiate 
between the existence probability before performing 
the statistical processing and the existence 
15 probability after performing the statistical 
processing, it is assumed that the existence 
probability before performing the statistical 
processing, that is, the existence probability p 3 
obtained by the equation 43 and the equation 44 is 
20 regarded as an evaluation reference value Vj. Then, 
it is assumed that a value obtained by performing 
the statistical processing on the evaluation 
reference value Vj is the existence probability (Jj- 

In the statistical processing performed on 
25 the evaluation reference value v 3 obtained by the 

equation 43 and the equation 44, first, as shown in 
Fig.26A, for example, a distribution function p(l) 
of the existence probability is obtained by applying 
the probability density distribution of the 
30 existence probability of the object to the 

distribution of the evaluation reference value v 3 . 
At this time, when the probability density 
distribution complies with normal distribution 
(Gaussian distribution) , for example, the 
35 distribution function p(l> of the existence 

probability can be represented by the following 
equation 45. 
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[Equation 45] 

V2;rcr 

In the equation, u indicates an average value, and a 
indicates a parameter indicating dispersion. Using 
5 the estimation reference value v jr estimation can be 
performed as shown in the following equation 46 and 
the equation 47. 
[Equation 46] 

M 
7=1 

10 [Equation 47] 

M . v 2 

After the distribution function p(l) of the 
existence probability is obtained in such a way, the 
existence probability (Jj is determined using the 
15 equation 48, for example. 
[Equation 48] 

fij - £ P<!)* 

in the equation, lj" and lj + respectively are an 
lower limit value and an upper limit value of a 

20 distance at which the object surface is regarded to 
exist on a projection plane Lj at a distance of lj . 
The values lj" and lj + can be provided by the 
following equation 4 9 and the equation 50, for 
example. 

25 [Equation 49] 



2<r 2 
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00 



h 2 • 1 j 

[Equation 50] 

Based on the existence probability Vj obtained by 

5 using the relationships of the equations 45-50, by 
displaying the three-dimensional image of the object 
on the DFD or by displaying the two-dimensional 
image viewed from an arbitrary viewpoint, an image 
in which noise effect of the original images is 

10 reduces can be displayed, wherein the original 

images are the images obtained by taking the object 
from the viewpoints Ci- 

In addition, as shown in Figs,24A and 24B, 
for example, when an object different from one 

15 appearing at other corresponding points appears in a 
corresponding point G if j corresponding to a 
projection point Tj due to occlusion of the object, 
it can be considered that relatively reliable 
estimation can be performed by obtaining the 

20 correlation degree Qj by excluding the corresponding 
point. Considering the case shown in Fig.24A, 
according to the method described so far, when 
obtaining the correlation degree Q' m of the 
projection point T f » on the line lp' extending from 

25 the observer's viewpoint P, the corresponding point 
Gi, m at which the surface of the object B, not object 
A, appears is also used. Therefore, the correlation 
degree Q'm obtained from the equation 40 becomes 
large so that it is difficult to estimate a distance 

30 at which the surface of the object exists on the 
line lp' . 

Thus, for example, as shown in Fig.27A, 
the correlation degree Q m for the projection point T ro 
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is obtained by excluding the corresponding point Gi, m 
at which the surface of the object B appears. At 
this time r in the example shown in Fig.27A, the 
correlation degree Q m is obtained using 
5 corresponding points G i+ i,a and Gi + 2,n corresponding to 
the projection point T tt . By doing so, since points 
very close to the surface of the object A appear on 
the corresponding points Gi+i,* and Gj. + 2, m r 
distribution shown in Fig.27B is obtained when 
10 obtaining the correlation degree Q'j from the 

equation 40, in which only the correlation degree 
Q' m of the projection point T' m is smaller than 
correlation degrees of other projection points. 
Therefore, effects of the occlusion can be decreased 
15 so that a three-dimensional shape close to an actual 
surface shape of the object can be estimated. 

In addition, at this time, several groups 
of sets of S that is a set of corresponding points 
Gi,j (viewpoints d) are set for obtaining the 
20 correlation degree Q 3 , and the correlation degree Qj 
of each projection point Tj on the line lp extending 
from the observer' s viewpoint P is obtained for 
every set of S. Then, the results are integrated so 
as to obtain a final existence probability. 
25 Generally, assuming that a group of corresponding 

points G±,j (viewpoints C L ) is 2 h (h^H), distribution 
function of the existence probability on the line lp 
obtained by using each group S h is Ph(l)* and that 
the existence probability obtained by each 
30 distribution function p h (D is 3j,h/ the integrated 
existence probability Pj can be obtained by the 
following equation 51. 
[Equation 51] 
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The color information Kj of the projection point Tj 
can be obtained using the following equation 52, for 
example, from color information K jth and existence 
probability B 3 ,h obtained for each group S h . 
[Equation 52] 



Accordingly, as shown in Fig. 28, for example, a 
distribution function p h (D/ obtained by a group S h , 
by which reliability for estimating the distance to 
the object surface is high and a clear peak appears 
is not susceptible to another distribution function 
p h (l), obtained by another group 3 h , by which 
reliability for estimating the distance to the 
15 object surface is low. Therefore, reliability for 
estimating the distance to each point of the object 
from the observer's viewpoint improves as a whole. 
(Embodiment 2-1) 

Figs. 29-34 are schematic diagrams for 
20 explaining an image generation method of the 

embodiment 2-1 of the present invention. Fig. 29 is 
a flow diagram showing an example of a general 
processing procedure. Fig. 30 is a flow diagram 
showing an example of a processing procedure of 
25 steps for determining color information and 

existence probability of projection points in Fig. 29 
Fig. 31 is a flow diagram showing an example of steps 
for determining the existence probability in Fig. 30. 
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Fig- 32 is a diagram showing an example foe setting a 
camera set. Figs. 33, 34A and 34B are diagrams for 
explaining a method for converting information of 
the projection plane to information of display plane, 
5 The image generation method of this 

embodiment 2-1 is a method in which images taken 
from plural viewpoints are used, a three-dimensional 
shape of the object appearing on the images is 
obtained, and each two-dimensional image to be 
10 displayed on each image display plane of the image 
display means having plural image display planes 
such as the DFD is generated, for example, based on 
the obtained three-dimensional shape of the object. 
The image generation method includes, as 
15 shown in Fig. 29 for example, a step 1 for obtaining 
the images of the object taken from the viewpoints 
C ± , a step 2 for setting the observer's viewpoint P, 
a step 3 for obtaining the three-dimensional shape 
of the object, a step 4 for converting color 
20 information and existence probability of each point 
(projection point) on the projection planes 
representing the obtained three-dimensional shape to 
color information and existence probability of each 
point (display point) on the image display planes so 
25 as to generate the two-dimensional images displayed 
on the image display planes, and a step 5 for 
displaying the display points on the image display 
planes with brightness or transparency according to 
the color information and the existence probability. 
30 The step 3 includes, as shown in Fig. 29 

for example, a step 301 for setting projection 
planes Lj of a multi-layered structure, a step 302 
for determining a reference viewpoint for obtaining 
the three-dimensional shape of the object, a step 
35 303 for setting projection point series each 

including a group of projection points T 3 , on the 
projection planes Lj , overlapping when viewed from 
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the reference viewpoint and for setting 
corresponding points Gi,j, on each obtained image, 
corresponding to each projection point T 3 of the 
projection point series, a step 304 for determining 
a group (to be referred as camera set) S h of 
viewpoints C L for obtaining correlation degrees Qj of 
each projection point T 3 , a step 305 for keeping an 
array for storing color information and existence 
probability of the projection points T 3 , and a step 
306 for determining the color information and the 
existence probability of the projection points Tj . 

In addition, as shown in Fig. 30 for 
example, the step 306 includes a step 30601 for 
initializing the projection point series, a step 
15 30602 for initializing the camera set S h and voting 
data, a step 30603 for initializing the projection 
point T 3 on the projection point series, a step 
30604 for determining the color information of the 
projection point T 3 , a step 30605 for calculating a 
20 correlation degree Q j/h using corresponding points 
included in the camera set S h among corresponding 
points G i(j for the projection point T 3 , a step 30606 
for repeating the step 30604 and the step 30605 for 
every projection point Tj on the projection point 
25 series to be processed, a step 30607 for voting each 
correlation degree Qj, h obtained for the camera set 
2 b , a step 30608 for updating the camera set S h , and 
repeating processing from the step 30 604 to the step 
30607 for every camera set, a step 30609 for 
determining existence probability of each projection 
point Tj based on the correlation degree Qj, h voted in 
the step 30 607, and a step 30610 for updating the 
projection point series and for repeating processing 
from the step 30602 to the step 30609 for every 
35 projection point series. 

In addition, as shown in Fig. 31, the step 
30609 includes a step 30609a for initializing the 
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camera set E h/ a step 30609b for calculating the 
evaluation reference value v j>h from the correlation 
degree Qj, h obtained by using the camera set S h , a 
step 30609c for determining the distribution 
5 function p h (l) of the existence probability by 

performing statistical processing of the evaluation 
reference value v j#h , a step 30609d for determining 
the existence probability p lfh of each projection 
point Tj from the distribution function of the 
10 existence probability, a step 30609e for updating 
the camera set 2 h and repeating processing from the 
step 30609b to the step 30609d, and a step 30609f 
for determining the existence probability |Jj of each 
projection point Tj by integrating the existence 
15 probability Pj.h obtained for each camera set S h - 

When generating the two-dimensional images 
to be displayed on each image display plane of the 
DFD, for example, using the image generation method 
of this embodiment 2-1, images are obtained by taken 
20 the object by cameras placed at plural different 

viewpoints C t { i=l , 2 , N) (step 1), first. At this 
time, the viewpoint Ci is a placement position of a 
camera for taking the image. For example, as shown 
in Fig. 21, it is assumed that the cameras are 
25 arranged one-dimensionally on a line. The 

arrangement of the viewpoints Ci of the cameras is 
not limited on the line, but the viewpoints may be 
arranged on plural lines or on a curve one- 
dimensionally. In addition, the viewpoints may be 
30 arranged like two-dimensional lattice on a plane or 
a curved surface instead of one-dimensionally. 
Although color images or black and white images can 
be obtained as the obtained images, color images in 
which each point (pixel) on the image is represented 
35 by color information using three primary colors of 

red (R) , green (G) and blue (B) are obtained in this 
embodiment . 
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Next, the observer's viewpoint P from 
which the three-dimensional image of the object to 
be displayed on the DFD is observed is set on a 
virtual space in an image generation apparatus such 
5 as a computer and the like (step 2) . 

Next, the three-dimensional shape used for 
generating the image is obtained (step 3) . In the 
step 3, first, projection planes Lj are set in the 
virtual space for estimating the three-dimensional 
10 shape (surface shape) of the object (step 301) . At 
this time, the projection planes Lj are set as 
planes parallel to the XY plane as shown in Fig. 21, 
for example. The setting intervals of the 
projection planes L 3 may be or may not be set to be 
15 the same as the intervals of the image display 

planes of the DFD on which the images are displayed. 

Next, the reference viewpoint for 
obtaining the three-dimensional shape of the object 
is determined (step 302). The reference viewpoint 
20 may be the observer's viewpoint, or may be any point 
on the three-dimensional space other than the 
observer's viewpoint. 

Next, projection point series each being a 
group of the projection points Tj, on each 
25 projection plane L 3 , overlapping when viewed from 
the observer's viewpoint P or the reference 
viewpoint, and corresponding points G ir j, on the 
obtained images, corresponding to each projection 
point T 3 are set (step 303) . 
30 The projection point Tj is represented as 

a point (Xj,Yj,Zj) on the virtual space (three- 
dimensional space) , and the coordinates of the 
corresponding point G i( j are represented as (Xi,j,yi.j) 
considering a two-dimensional xy coordinate system 
35 on the image plane of the image taken from the 
viewpoint Ci. The coordinates (xi,j,yi,j) of - the 
corresponding point G i(J are obtained by projecting 
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the projection point (X Jf Y Jr 2j) to the image plane of 
the image taken from the viewpoint Ci- The general 
3 rows and 4 columns conversion matrix described in 
the first embodiment can be used for the projection. 
5 When using the image generation apparatus 

like a computer, the image to be processed is a so- 
called digital image, and is represented by a two- 
dimensional array in the memory of the apparatus. 
In the following, the coordinate system representing 
10 the position in the array is referred to as digital 
image coordinate system, and the position is 
represented as (u,v) . For example, considering a 
digital image of lateral 640 pixels and vertical 480 
pixels, a position of each pixel of the digital 
15 image can be represented by an argument u that may 
take any integer from 0 to 639 and an argument v 
that may take any integer from 0 to 479. The color 
information of the point can be provided as data 
obtained by quantizing information of red (R) , green 
20 (G) and blue (B) at the address into 8 bits, for 
example . 

The coordinates (Xi,j,yi,j) of the 
corresponding point G ifJ in the three-dimensional 
virtual space are in a one-to-one correspondence 
25 with the digital image coordinate system (u,v). For 
example, the relationship shown in the equation 53 
holds true. 
[Equation 53] 
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In the equation 53, the u axis of the digital image 
coordinate system is set parallel to the x axis. In 
addition, in the equation 53, k u and ^respectively 
represent unit lengths of the u axis and the v axis 
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of the digital image coordinate system with respect 
to the (x,y) coordinate system on the virtual space, 
and 6 indicates an angle formed between the u axis 

and the v axis. 
5 Thus, in the step 303, the coordinates 

(Xj,Y jf Zj) of the projection point T 3 are associated 
with the digital image coordinates (ui^vij) . The 
establishment of the association may be performed by 
providing values of (X^Y^Zj) in a table for every 
10 (u tj ,Vij), or by setting values of (Xj,Yj,Zj) only for 
representative values of (u i:) ,v i3 ) so as to obtain 
other points by performing interpolation processing 
such as linear interpolation, for example. 

In addition, although (u,v) take discrete 
15 values in the digital image coordinate system, it is 
assumed that <u,v) take continuous values unless 
otherwise specified in the following descriptions, 
and it is assumed to perform proper discretization 
processing when accessing the two-dimensional array. 
20 Next, after performing processing of the 

step 303, groups (camera set) E h of viewpoints Ci of 
the cameras used for obtaining the correlation 
degree Qj is determined (step 304). When the 
viewpoints Ci, that is, the cameras are placed like 
25 a 3*3 lattice as shown in Fig. 32, for example, four 
camera sets S h are determined as Hi={ C%, C 2 , C3, C 5 ) , 
22={C 3 ,C 5 ,C6,C9), 33={C 5 ,C7,Cb,C 9 } and S 4 =(Ci, C 4 , C 5 , C 7 } . 

Any method can be adopted as the method 
for determining the camera sets 2 h . In the example 
30 shown in Fig. 32, the sets are not limited to the 
four sets Hi, S 2 , H 3 and S 4 , so that other camera 
sets may be provided. In addition, the camera sets 
S h may be provided beforehand according to the 
placement status of the cameras (viewpoints C±) , or 
35 may be specified by the observer. 

Next, after performing the processing of 
the step 304, an array is kept, in a memory (storing 
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means) of the image generation apparatus, for 
storing information of the color information K 4 of 
the projection point T 3 and the probability (Jj that 
the object exists (step 305) . As to the array for 
5 storing the information, an array is kept such that 
8 bits are assigned for each pieces of information 
K 3 and at the projection point T 3 that are color 
information of red ( R) , green (G) and blue (B) and 
the probability that the object exists- 
10 Next, after performing the processing of 

the step 305, the color information and the 
probability that the object exists at each 
projection point T 3 are determined using the 
obtained plural images (step 306) . In the step 306, 
15 for a projection point series, processing is 

performed for obtaining the color information K 3 , h 
and the correlation degree Q jfh at each projection 
point T 3 on the projection point series using the 
specified camera set S h . Then, the processing is 
20 repeated for every camera set S h . Then, the 
repeating processing is repeated for every 
projection point series. Thus, in the step 306, the 
projection point series is initialized as shown in 
Fig. 30, first (step 30601). 
25 Next, the camera set Sh and voting data of 

correlation degree are initialized (step 30602) . 

Next, the projection point T 3 on the 
projection point series to be processed is 
initialized to j=l, for example (step 30603). 
30 Next, the color information Kj, h of the 

projection point T 3 is determined based on the color 
information of the corresponding points G i<3 included 
in the selected camera set S h (step 30604) . At this 
time, color information K 3 , h of the projection point 
35 Tj is determined as an average of color information 
K ifj of the corresponding points Gi, 3 included in the 
camera set S h , for example. 
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Next, the correlation degree Q j/h of the 
corresponding points G ifj included in the selected 
camera set 3 n is calculated (30605). At this time, 
the correlation degree Q jrh is calculated using the 
5 equation 40, for example. 

Next, the projection point Tj is updated, 
and it is determined whether processing of the steps 
30604 and 30605 is performed for every projection 
point on the projection point series to be processed 
10 (step 30606) . When a projection point on which the 
processing of the steps 30604 and 30605 has not been 
performed remains, the processing is repeated by 
returning to the step 30604. 

After the processing of the steps 30604 
15 and 30605 is performed for every projection point on 
the projection point series to be processed, the 
results, that is, the color information Kj, h and the 
correlation degree Qj, n obtained from the 
corresponding points Gi,j included in the selected 
20 camera set S h are voted (step 30607) * 

Next, after completing the processing of 
the step 30607, the camera set S h is updated, and it 
is determined whether there is a camera set for 
which processing of steps 30604-30607 has not been 
25 performed for the projection point series to be 

processed (step 30608) . When there is a camera set 
for which processing of steps 30604-30607 has not 
been performed, the processing is repeated by 
returning to the step 30603. 
30- After performing the processing of steps 

30604-30607 for every camera set S h for the 
projection point series to be processed, the color 
information Kj and the existence probability (3j of 
the projection points Tj are determined from the 
35 color information K jfh and the correlation degree Q 3 , h 
that are voted at the step 30607 (step 30609) , 

In the step 30609, the camera set S h is 
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initialized first as shown in Fig. 31, for example 

(step 30609a) . 

Next, the evaluation reference value v j/h 
is calculated from the correlation degree Q 3 ,h of 
5 each projection point T a calculated using the camera 
set 2 h (step 30609b) . The evaluation reference 
value v 3(h is obtained by performing conversion 
processing represented by the equation 43 and the 
equation 44, for example. 
10 " Next, statistical processing for the 

evaluation reference value v jfh is performed, so that 
the distribution function p»(l) of the existence 
probability when the camera set 2 h is used is 
obtained (step 30609c) . The distribution function 
15 p h (l) is obtained using the equation 45, the 
equation 46 and the equation 47, for example. 

Next, based on the distribution function 
Ph (l) of the existence probability in the case when 
the camera set S h is used, the probability 
20 (existence probability) p jfh that the surface of the 
object exits on each projection point T 3 is 
determined (step 30609d) . The existence probability 
Pi, h is determined using the equation 48, the 
equation 49 and the equation 50, for example. 
25 Next, the camera set 3 h is updated, and it 

is determined whether there is a camera set 3 for 
which processing from the step 30609b to the step 
30609d has not been performed (step 30609e) . When 
there is a camera set S for which processing from 
30 the step 30609b to the step 30609d has not been 

performed, the processing is repeated by returning 
to the step 30609b. 

After performing the processing from the 
step 30609b to the step 30609d for the projection 
35 point series to be processed, the processing results 
are integrated so that the color information Kj and 
the existence probability 3j a ^ the projection point 
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Tj are determined (step 30609f) . At this time, the 
color information K 3 is obtained using the equation 
52, for example. The existence probability is 
obtained using the equation 51, for example. 

5 After completing the processing of the 

step 30609f, the processing of the step 30609 
completes. Then, next, the projection point series 
is updated, and it is determined whether there is 
projection point series for which processing from 
10 the step 30602 to the step 30609 has not been 

performed (step 30610) . When there is projection 
point series for which processing from the step 
30602 to the step 30609 has not been performed, the 
processing is repeated by returning to the step 

15 30602. 

After performing processing from the step 
30602 to the step 30609 for every projection point 
series, processing of the step 306 (step 3) 
completes so that the three-dimensional shape of the 
20 object can be obtained. 

Next, after the processing of the step 3 
completes and the three-dimensional shape of the 
object is obtained, two-dimensional images to be 
displayed on each image display plane of the DFD are 
25 generated based on the three-dimensional shape of 
the object. For generating the two-dimensional 
image, image generation planes LD r (r=l, 2, R) for 
generating the two-dimensional images are set on the 
virtual space from which the shape of the object is 
30 obtained as shown in Fig. 33, for example. 

First,/ a case is considered in which the 
setting number of the projection planes L, and the 
intervals are the same as the number of the image 
generation planes LD r and the intervals respectively. 
35 In this case, the color information KD h and the 

existence probability Yr at a display point A r on the 
image generation plane LD r are determined to be the 
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color information Kj and the existence probability 
(jj at the projection point Tj the same as the display 
point A x - 

The setting intervals of the projection 
5 planes Lj are not necessarily the same as the 

placement intervals of the image generation planes 
LD r , and the setting number of the projection planes 
L 3 is not necessarily the same as the setting number 
of the image generation planes LD t . That is, 
10 according to the setting method of the projection 
planes Lj, there is a case where the setting 
intervals of the projection planes Lj are not the 
same as the placement intervals of the image 
generation planes LD E as shown in Fig.34A, for 
15 example. In such a case, the color information KD t 
and the existence probability Yr at the cross points 
(display point) A r of the line lr extending from the 
observer's viewpoint P and each image generation 
surface LD r are obtained according to the following 
20 procedure. 

First, the color information KD r at each 
display point A r is determined to be an average 
value of color information K of projection points T, 
that is the projection points T-j on the line lp, for 
25 which the display point A r (image generation plane 
LD r ) becomes the nearest display point (image 
generation plane) - Instead of the average value, 
the color information KD r at the display point A r may 
be determined as color information K of a projection 
30 point T nearest to the display point A r . 

On the other hand, the existence 
probability of the display point A E is determined as 
a value obtained by adding existence probabilities p 
of projection points T for which the display point 
35 A r (image generation plane LD £ ) becomes the nearest 
display point (image generation plane) . Assuming 
that a set of projection planes L 3 for which an 
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image generation plane LD r becomes the nearest image 
generation plane is {LjljSR), the existence 
probability Yh of the display point A r on the image 
generation plane LD r can be obtained by the 
5 following equation 54 using the existence 

probability p a of the projection points Tj of each 
projection plane Lj - 
[Equation 54] 

10 Considering the case shown in Fig.34A, the image 
generation plane LD X becomes the nearest image 
generation plane for the projection planes la, L 2 
and L 3 . Therefore, the color information KD r of the 
display point A r is determined to be an average 
15 value of the pieces of color information K i# K 2 and 
K 3 of the projection points Ti, T 2 and T 3 / for 
example. In addition, the existence probability Y r 
of the display point A r is determined to be a sum of 
the existence probabilities pi, p 2 and P 3 of the 
20 projection points T lf T 2 and T 3 , for example. In the 
same way, the color information KD 2 of the display 
point A 2 on the image generation plane LD 2 is 
determined to be an average value of the pieces of 
color information K 4 and K s of the projection points 
25 T 4 and T 5 , for example. In addition, the existence 
probability y 2 of the display point A 2 is determined 
to be a sum of the existence probabilities p 4 and p s 
of the projection points T 4 and T 5 , for example. 

In addition, for example, when the setting 
30 intervals of the image generation planes D r are not 
the same as the setting intervals of the projection 
planes Lj so that two projection planes L x and L 2 are 
set between two successive image generation planes 
LD X and LD 2 as shown in Fig.34B, for example, the 
35 existence probability Yi, Y2 of the display point A x , 
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A 2 of the image generation plane LDi, LD 2 may be 
obtaining by distributing the existence probability 
(5 3 of the projection point T 3 of the projection plane 
Lj according to the ratio of distance to each image 
generation plane LD lf LD 2 . Generally, assuming that 
a set of plural projection planes L 3 between the 
image generation planes LDi and LD 2 is {LjljSj}, the 
existence probability Yr of the display point A E on 
the image generation plane LD t can be supplied by 
the following equation 55 using the existence 
probability Bj of each projection point Tj . 
[Equation 55] 



jeJ 

wherein w jft indicates a coefficient representing a 
15 degree of contribution of the projection plane Lj to 
the image generation plane LD E . 

Here, it is considered that projection 
planes Li and L 2 are set between the two image 
generation planes LDi and LD 2 as shown in Fig.34B. 
20 At this time, when the distances from the projection 
plane Li to each of the image generation planes LDi 
and LD 2 are B x and B 2 respectively, the degrees w lrl 
and wi, 2 of contribution of the projection plane L x to 
the image generation planes LDj and LD 2 are provided 
25 respectively by the following equation 56. 
[Equation 56] 

In the same way, when the distances from the 
projection plane L 2 to each of the image generation 
30 planes LDi and LD 2 are B 3 and B< respectively, the 
degrees w 2 ,i and w 2 , 2 of contribution of the 
projection plane L 2 to the image generation surfaces 
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LDi and LD 2 are provided respectively by the 
following equation 57 . 
[Equation 57] 

5 As a result, the existence probability yi 

of the display point A x of the image generation 
plane LDi and the existence probability yz of the 
display point A 2 of the image generation plane hT> 2 
can be supplied respectively by the following 
10 equation 58. 

[Equation 58] 

According to the above-mentioned procedure, 
by performing the processing of the step 4, the two- 
15 dimensional images to be displayed on each image 
display plane of the DFD can be obtained. Then, 
each point (pixel) oh each image display plane of 
the DFD is displayed with color information A that 
is assigned to each point on each image generation 
20 plane LD (step 5). At this time, when the DFD is 
the brightness modulation type, color information 
KD r of each display point A r of each image generation 
plane LD r is displayed with brightness according to 
the existence probability y K . When the DFD is the 
25 transmission type, transparency of each display 
point A* is set to transparency according to the 
existence probability Yr and it is displayed. 

Accordingly, by performing the processing 
from the step 1 to the step 5 shown in Fig. 29 so as 
30 to display the three-dimensional image of the object 
on the DFD, a three-dimensional image that looks 
natural for the observer can be displayed without 
obtaining the accurate three-dimensional shape of 
the object as described in the principle. 



2005#12fll6B 2ie§53# 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 114 



-104- 



Figs. 35-37 are schematic diagrams showing 
general configurations of an apparatus and a system 
to which the image generation method of this 
embodiment 2-1 is applied. Fig. 35 is a block 
5 diagram showing a configuration example of the image 
generation apparatus. Fig. 36 is a diagram showing a 
configuration example of an image display system 
using the image generation apparatus. Fig. 37 is a 
diagram showing another configuration example of the 
10 image display system using the image generation 
apparatus . 

In Fig. 35, 6 indicates the image 
generation apparatus, 601 indicates an object image 
obtaining means, 602 indicates a reference viewpoint 
15 setting means, 603 indicates a projection plane 
setting means, 604 indicates a projection plane 
information storing area keeping means, 605 
indicates a color information/existence probability 
determination means, 606 indicates a projection 
20 plane information - display plane information 

conversion means, 607 indicates an image output 
means, 7 indicates an image display means (DFD) , 8 
indicates an object image taking means, and 9 
indicates an observer viewpoint input means - 
25 As shown in Fig. 35, for example, the image 

generation apparatus 6 to which the image generation 
method of this embodiment 2-1 is applied includes 
the object image obtaining means 601 for obtaining 
plural object images taken under different 
30 photographing conditions, the reference viewpoint 
setting means 602 for setting a viewpoint of the 
observer who observes a generated image, the 
projection plane setting means 603 for setting 
projection planes, projection point series, 
35 projection points, camera sets and the like for 
determining the existence probability, the 
projection plane information storing area keeping 
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means 604 for keeping an area for storing color 
information and existence probability of points 
(projection points) on the projection plane, the 
color information/existence probability 
determination means 605 for determining the color 
information of the projection points and the 
probability (existence probability) that the object 
exist at the projection points, the projection plane 
information - display plane information conversion 
means 606 for converting the color information and 
the existence probability information of the 
projection point to color information and existence 
probability of the display plane, and the image 
output means 607. An image output from the image 
15 output means 607 is displayed by the image display 

means 7 such as the DFD including plural overlapping 
display planes. 

The object image obtaining means 601 
obtains images of the object taken by the object 
20 image taking means (camera) 8. The image to be 

obtained may be one directly obtained from the image 
taken by the object image taking means 8, or may be 
indirectly obtained from a recording medium that is 
magnetic, electrical or optical that stores the 
25 image taken by the object image taking means 8. 

The observer viewpoint setting means 602 
sets relative position relationship between the 
observer's viewpoint P and the image generation 
planes LD r such as distance from the viewpoint P to 
30 the image display means 7 and the line of sight 

based on information that is input by the observer 
using the image condition input means 9 such as the 
mouse or a keyboard, for example. The image 
condition input means 9 may be a means for detecting 
35 posture or line of sight of the observer so as to 
input information according to the posture or the 
line of sight. 
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The projection plane etc. setting means 
603 sets parallel projection planes Lj in which the 
distances from the projection planes Lj from the 
viewpoint (camera) dare lj ( j=l, 2, «, M) ■ In addition, 
5 the projection plane setting means 603 also sets 
projection point series each being a group of 
projection points Tj on each projection plane Lj 
overlapping when viewed from the observer's 
viewpoint P set by the observer viewpoint setting 
10 means 602, and the corresponding points G iljf on each 
image, corresponding to the projection point Tj ♦ In 
addition, the projection plane setting means 603 may 
set camera sets S h based on the condition input by 
the image condition input means 9. 
15 The projection plane information storing 

area keeping means 604 keeps the area for the color 
information Kj and the existence probability Pj of 
each projection point Tj on each projection plane in 
a memory provided in the apparatus, for example. 
20 The color information/existence 

probability determination means 605 determines the 
color information Kj of the projection point Tj from 
color information of corresponding points Gij on the 
images corresponding to the projection point T 3 , and 
25 determines the probability (Jj that the surface of 
the object exists on the projection point Tj based 
on the before-mentioned principle. 

The projection plane information - display 
plane information conversion means 606 converts the 
30 color information and the existence probability of 
the projection plane into color information and the 
brightness distribution ratio of a point on the 
plane on which the image to be displayed on the 
display plane of the image display means 7 is 
35 generated, that is, converts the color information 
and the existence probability of the projection 
plane into color information and the brightness 
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distribution ratio of a point on the image 

generation plane. 

The image generation apparatus 6 performs 
processing from the step 1 to the step 5 described 
in this embodiment 2-1 to generate the image to be 
displayed on the DFD. That is, it is not necessary 
for the image generation apparatus 6 to perform 
processing for obtaining an accurate three- 
dimensional shape of the object as conventionally 
performed. Therefore, even by an apparatus that 
does not have high processing ability, the images to 
be displayed on the DFD can be generated easily and 

at high speed. 

The image generation apparatus 6 can be 

15 also realized by a computer and a program executed 
by the computer, for example. In such as case, a 
program that describes instructions corresponding to 
the processing procedure described in this 
embodiment 2-1 is executed on the computer. In 

20 addition, the program can be provided by storing in 
a recording medium that is magnetic, electrical or 
optical, or can be provided via a network such as 

the Internet. 

In addition, the image display system 

25 using the image generation apparatus 6 can be 

considered to be configured as shown in Fig. 36, for 
example. In the configuration, the object image 
taking means 8 may be placed at a place near the 
space where the observer User observes the image 

30 display means (DFD) 7, or may be placed at a place 
geographically apart. When the object image taking 
means 8 is placed at the geographically apart place, 
the taken images can be transferred to the image 
generation apparatus 6 using a network such as the 
35 Internet. 

In addition, as shown in Fig. 36, the image 
display system using the image generation apparatus 
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6 can be applied not only to a case where an 
observer User observes an object Obj but also to a 
bi-directional communication system such as TV phone 
and TV conference. In the case, spaces in which the 
5 observers UserA and UserB exist are respectively 

provided with image generation apparatuses 6A and 6B, 
image display means (DFDs) 7A and 7B, object image 
taking means 8A and 8B, and reference viewpoint 
setting means 9A and 9B as shown in Fig. 37, for 
10 example- Then, for example, by connecting between 
the image generation apparatuses placed at the 
spaces where the observers UserA and UserB exist by 
the network 10 such as the Internet, the observer 
UserA can observe, by the image display means 7A, 
15 the three-dimensional image of the observer UserB 

generated from the images taken by the object image 
taking means 8B, In the same way, the observer 
UserB can observe, by the image display means 7B, 
the three-dimensional image of the observer UserA 
20 generated from the images taken by the object image 
taking means 8A. 

In addition, for applying the image 
generation apparatuses to such bi-directional 
communication system, it is not necessary that both 
25 of the image generation apparatuses 6A and 6B are 

configured as those shown in Fig. 35. Either one of 
the image generation apparatuses 6A and 6B can be a 
general communication terminal that does not include 
the configuration means shown in Fig. 35. In 
30 addition, each configuration means shown in Fig. 35 
may be distributed to the image generation 
apparatuses 6A and 6B. 

In addition, as shown in Fig. 37, if 
another image generation apparatus 6C is provided on 
35 the network 10, the three-dimensional image of the 
object to be displayed on the image display means 
(DFD) 7A and 7B can be obtained using the image 
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generation apparatus 6C on the network 10 even when 
the image generation apparatuses 6A and 6B are not 
placed at the spaces where the observers UserA and 
UserB exists. 
5 In addition, in the image generation 

system shown in Fig. 37 , although the number of the 
users UserA and UserB are two, the image generation 
scheme can be applied to a communication system 
among more observers (users) . 
10 In addition, although Figs. 36 and 37 show 

that the object image taking means includes four 
cameras, the number of the cameras may be two or 
three or more than four. In addition, the cameras 
may be placed one-dimensionally on a line or a curve, 
15 or placed like a two-dimensional lattice on a plane 
or a curved surface. 

As described above, according to the image 
generation method of this embodiment 2-1, the three- 
dimensional image that looks natural for the 
20 observer can be displayed without obtaining an 

accurate three-dimensional shape of the object to be 
displayed . 

In addition, in the image generation 
method of this embodiment 2-1, although processing 

25 is performed by setting the camera sets S h in step 
304 beforehand, the processing is not limited to 
this way. The camera sets may be dynamically set by 
software processing such that the sets matches 
conditions specified by the observer while 

30 performing processing for generating the images to 
be displayed, for example. In this case, by 
inputting the conditions such as distribution or 
threshold of the correlation degree Qj from the 
image condition input means by the observer and by 

35 performing processing of the step 306 while 

searching for camera sets that match the conditions, 
it can be considered that the three-dimensional 
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image close to an image desired by the observer can 
be displayed. 

In addition, in the image generation 
method of this embodiment 2-1, although a case is 
5 described as an example in which a color image in 
which the point (pixel) is represented by color 
information using three primary color of red (R) , 
green (G) and blue (B) is obtained and the three- 
dimensional shape of the object is generated, the 

10 image to be obtained is not limited to the color 
image- In the image generation method of this 
embodiment 2-1, a black and white image in which the 
point (pixel) is represented by brightness (Y) and 
color-difference (U,V) may be obtained so as to 

15 obtain the three-dimensional shape of the object . 
In the case when the image to be obtained is the 
black and white image, the three-dimensional shape 
can be obtained, by the procedure described in this 
embodiment 2-1, using brightness information (Y) as 

20 information corresponding to the color information 

so that the two-dimensional images can be generated. 
(Embodiment 2-2) 

Fig. 38-42 are block diagrams for 
explaining an arbitrary viewpoint image generation 

25 method of the embodiment 2-2. Fig . 38 is a flow 

diagram showing an example of the whole processing 
procedure. Fig. 39 is a diagram for explaining a 
principle of rendering. Fig. 40 is a diagram for 
explaining problems that arise when generating the 

30 arbitrary viewpoint image- Figs.41A and B are 

diagrams for explaining a method for solving the 
problems that arise when generating the arbitrary 
viewpoint image. Fig. 42 is a flow diagram showing 
an example of the processing procedure for 

35 converting the existence probability to transparency. 

In the embodiment 2-1, although a method 
for generating the two-dimensional images to be 
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displayed on each image display plane of the- 
apparatus having plural image display planes like 
the DFD using the three-dimensional shape of the 
object obtained in the step 3 is taken as an example, 
5 the three-dimensional shape of the object is not 
limited to that, but it can be also used for 
generating two-dimensional images of the object 
viewed from arbitrary viewpoints, in this case, the 
difference compared to the embodiment 2-1 is that 

10 rendering is performed after the step 3, that is, 
processing of step 11 for generating the two- 
dimensional images of the object viewed from the 
observer's viewpoint based on the three-dimensional 
shape. Since the processing from the step 1 to the 

15 step 3 in which the three-dimensional shape of the 
object is obtained is the same as that described in 
the embodiment 2-1, it is not described in detail. 

In the step 11 for rendering in the 
arbitrary viewpoint image generation method of this 

20 embodiment 2-2, the color information of each point 
(pixel) on the arbitrary viewpoint image to be 
displayed is determined by performing mixing 
processing for color information Kj of projection 
points T} ( j-l f 2,.»,M) overlapping with a point A on 

25 the arbitrary viewpoint image as shown in Fig. 39, 

for example. In the mixing processing, for example, 
weight that is the existence probability (J-j is 
assigned to color information Kj of each projection 
point Tj, and weighted color information Kj are mixed, 

30 so that the color information K A of the point on the 
image to be generated is calculated by the following 
equation 59, for example. 
[Equation 59] 



M 
7=1 
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However, according to mixing processing using the 
equation 59, the color information of the point of 
the image may be far different from the color 
information of the actual surface of the object or 
5 may be not fall within an effective color space 
depending on the shape of the object or position 
relationship between the reference viewpoint R and 
the virtual viewpoint P, for example. Here, a case 
is considered in which two projection planes Li and 

10 L 2 , a reference viewpoint R and a virtual viewpoint 
P are applied for an actual object in position 
relationship as shown in Fig. 40. In this case, when 
existence probabilities p x and p 2 are determined with 
respect to the projection points T a and T 2 

15 overlapping when viewed from the reference viewpoint 
R by the method described in the embodiment 2-1, p a 
becomes almost 0 and p 2 becomes almost 1* Similarly, 
when existence probabilities p'i and p f 2 are 
determined with respect to the projection points T'i 

20 and T f 2 overlapping when viewed from the reference 
viewpoint R by the method described in the 
embodiment 2-1, p r i becomes almost 1 and p' 2 becomes 
almost 0. 

The color information K A at the point A of 
25 the image plane of the virtual viewpoint P is 
obtained by adding, after assigning weights 
according to the existence probabilities (S 1 x and p 2 , 
color information K'i and K 2 of the projection points 
T'i and T 2 overlapping with the point A on the image 
30 plane when viewed from the virtual viewpoint P. By 
doing so, both of them become almost 1, so that the 
color information K A of the point A becomes Ka^K'i+I^. 

However, when the object Obj is viewed 
from the virtual viewpoint P, since the projection 
35 point T»! is blocked by the projection point T 2 , 

actual color information at the point A on the image 
plane should be K A =K 2 . That is, brightness of each 
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component (R,G,B) of the generated color information 
K A at the point A on the image is increased by K'j 
compared with the actual color information. 

In addition, when each component of the 
5 color information K'i and K 2 of the projection points 
T'i and T2 has large brightness, the color 
information K A of the point A exceeds a limit of an 
effective color space. Therefore, clipping 
processing is necessary for the color information to 

10 fall within the limit of the effective color space. 

To solve the problem, transparency that 
may take plural gradations from transparent to 
opaque is provided to each projection point based on 
the existence probability of the projection point. 

15 In this case, the mixing processing is performed 
successively from a projection point of the 
generated image far from the viewpoint to a 
projection point near the viewpoint, in which color 
information obtained by the mixing processing up to 

20 a projection point is obtained by performing 

interior division between color information at the 
projection point and color information obtained by 
mixing processing up to a previous projection point 
with a ratio according to the transparency. The 

25 color information obtained by the mixing processing 
is interior division between color information at a 
stage and color information at the next stage. 

For explaining the principle of the mixing 
processing of the color information, assuming a case 

30 in which projection planes Lj ( j-1, 2, ...,M) , projection 
points Tj, and a vector Kj having components of red 
(R) , green (G) and blue (B) and representing color 
information of the projection point are set in a 
color space V as shown in Fig.4lA, for example. It 

35 is assumed that the color space V is represented by 
the following equation 60. 
[Equation 60] 
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Kj ,e V, V a { (R, G, B) |0 < R < 1, 0 < G < 1, 0 < B < 1 

In addition,, it is assumed that the transparency ofg 
of the projection point Tj is set so as to satisfy a 
5 condition shown in the following equation 61 . 
[Equation 61) 

o < a j ^ 1 

The color information D m obtained by mixing 
processing up to the argument j=ra is represented by 

10 recurrence formulas shown in equations 62 and 63. 

Then, color information D M of the projection plane L M 
nearest to the virtual viewpoint P, that is, color 
information D M obtained by performing mixing 
processing up to the argument j=M becomes the color 

15 information K A at the point A on the image plane of 
the image to be generated. 
[Equation 62] 

D m =a m K m +{\-a m )D m _ i 

[Equation 63 ] 

20 D l =oc x K x 

In addition, as shown by the relationship of the 
equations 61 and 62, the color information D m is a 
internally divided point between the vector K m and 
the color information D m . x in the color space V as 

25 shown in Fig.41B. Therefore, as to the color 

information D m , when K m ^V and D m -i^V are satisfied, 
D ffl ev holds true. 

Then, as described in the first embodiment 
when the conditions of the equations 60 and 61 are 

30 satisfied, D M e V is ensured with respect to the 
color information D M at the virtual viewpoint P. 

That is, by setting the color information 
Kj and the transparency a 3 at the projection point Tj 
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so as to satisfy the equations 60 and 61 f the color 
information D M of the point A of the image to be 
generated can always fall within a proper color 
space V. 

5 As mentioned above, for generating images 

obtained by viewing the same object from plural 
virtual viewpoints, by performing the mixing 
processing of the color information using the 
transparency a-j, all images can be generated within 

10 a limit of proper color information even when 
calculating the color information and the 
transparency of the projection point from a 
viewpoint (reference viewpoint) if the color 
information and the transparency satisfies the 

15 equations 60 and 61. 

Therefore, according to a generation 
method of the arbitrary viewpoint image of this 
embodiment 2-2, processing for converting the 
existence probability (i-j to the transparency aj is 

20 performed after the step 30609 shown in Fig. 30 or in 
the step 11 for performing rendering. 

In the processing for converting the 
existence probability (3j to the transparency o<j, the 
projection point Tj is initialized to j=M first as 

25 shown in Fig. 42, for example (step 1101). Next, p w 
is substituted to transparency a M of the projection 
point T M such that a M =P M (step 1102) . 

Next, the value of the argument j is 
updated to j=j-l (step 1103) . Then, it is 

30 determined whether the transparency cxj+i is 1 (step 

1104) . If transparency aj+i^l, the transparency <*j 
is obtained from the equation 64, for example (step 

1105) . 
[Equation 64] 
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171=7+1 

If the transparency or-j+i is 1, 1 is substituted into 
otj (aj = l), for example {step 1106). When obtaining 
the transparency QCj in the step 1105, the equation 
5 is not limited to the equation 64, and other 
equations can be used. In addition, although 
detailed descriptions are not provided, since any 
value can be substituted into the transparency otj 
actually in step 1106 r a value other than 1 may be 

10 substituted into the transparency aj . 

Next/ it is determined whether processing 
from the step 1104 to the step 1106 has been 
performed up to the argument j=l (step 1107). When 
the processing has not been performed, the 

15 processing is repeated after returning to the step 
1103. 

After performing the processing from the 
step 1104 to the step 1106 up to the argument j=l, 
processing for converting the existence probability 

20 Pj of the projection point Tj overlapping with the 
point A, on the image plane, viewed from the 
observer' s viewpoint P into the transparency otj 
completes- After that, mixing processing using the 
equation 62 and the equation 63 is performed. Then, 

25 by performing the processing for every point (pixel) 
on the arbitrary viewpoint image, the arbitrary 
viewpoint image from the observer's viewpoint P can 
be obtained. 

The basic configuration of the image 

30 generation apparatus for generating such arbitrary 
viewpoint image is the same as that of the image 
generation apparatus described in the embodiment 2-1. 
As a means corresponding to the projection plane 
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information - display plane information conversion 
means 606 shown in Fig. 35, a means for performing 
before-mentioned mixing processing may be provided. 
Thus, descriptions on the apparatus are not provided. 
5 As described above, according to the image 

generation method of this embodiment 2-2, the three- 
dimensional image that looks natural for the 
observer can be displayed without obtaining an 
accurate three-dimensional shape of the object to be 
10 displayed. 

In addition, also in the image generation 
method of this embodiment 2-2, the camera sets Eh 
may be dynamically set by programmatic processing 
such that the sets matche conditions specified by 

15 the observer while performing processing for 

generating the image to be displayed, for example. 
In this case, by inputting the conditions such as 
distribution or threshold of the correlation degree 
Qj from the image condition input means by the 

20 observer and by performing processing of the step 

306 while searching for camera sets that matche the 
condition, it can be considered that the three- 
dimensional image close to an image desired by the 
observer can be displayed. 

25 In addition, also in the image generation 

method of this embodiment 2-2, as to the image to be 
obtained, either of the color image or the black and 
white image can be adopted. In the case when the 
image to be obtained is the black and white image, 

30 after the three-dimensional shape is obtained by the 
procedure described in the embodiment 2-1 using the 
brightness information (Y) as information 
corresponding to the color information, the virtual 
viewpoint image can be generated according to the 

35 procedure described in this embodiment 2-2. 

(Effects of the second embodiment) 

As mentioned above, according to the image 
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generation method of the second embodiment, when 
obtaining the three-dimensional shape of the object, 
plural projection planes are set, and the 
probability (existence probability) that the surface 

5 of the object exists is provided for the points 
(projection point) of each projection plane 
overlapping viewed from the reference viewpoint. 
That is, the method does not intend to obtain an 
accurate three-dimensional shape of the object 
10 assuming that the surface of the object exists on a 
projection plane among projection points overlapping 
viewed from the reference viewpoint, as 
conventionally performed, but the three-dimensional 
shape of the object is obtained assuming that the 

15 surface of the object exists on each projection 

point with an existence probability- Accordingly, 
when estimating a distance of the surface of the 
object in a direction from the reference viewpoint, 
a portion in which the estimation reliability is low 

20 is rendered vaguely in a ratio according to the 
existence probability of each projection point. 
Therefore, when generating the image based on the 
three-dimensional shape of the object, discontinuous 
noise that appears when the estimation of the 

25 distance of the object surface includes an error, 
that occurs in a conventional technology, can be 
made inconspicuous, so that an image that looks 
natural can be generated. 

In addition, as mentioned before, when the 

30 object surface near a projection point does not 

appear from a viewpoint of an area due to an effect 
of the occlusion when calculating the existence 
probability based on plural images included in the 
camera sets from obtained images, existence 

35 probability in which the image taken from the 

viewpoint is excluded can be calculated so that the 
reliability of the existence probability of each 
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projection point increases* 

In addition, when determining the 
existence probability, if the probability density 
distribution of the existence probability can be 
5 estimated to some extent, the evaluation reference 
value may be calculated from the correlation degree 
of each projection point so as to determine the 
existence probability based on the distribution 
function obtained by performing statistical 
10 processing on the evaluation reference value. 
Accordingly, by determining the existence 
probability by performing the statistical processing, 
deterioration of the reliability due to the noise on 
the obtained images can be prevented. 
15 [Third embodiment] 

Next, the third embodiment of the present 
invention is described. The third embodiment is an 
embodiment mainly corresponding to claims 22-29. In 
the third embodiment, based on plural images (multi- 
20 focus image) taken by changing focusing distance 

from one viewpoint, the three-dimensional shape of 
the object appearing in the images is obtained, so 
that the image of the object viewed from an 
arbitrary viewpoint (virtual viewpoint) is generated. 
25 That is, compared to the first and second 

embodiments in which the three-dimensional shape of 
the object is obtained based on plural images of the 
object taken from plural viewpoints, this embodiment 
is characterized by using plural images taken by 
30 changing focusing distance from one viewpoint. Also 
in this embodiment, the three-dimensional shape of 
the object is represented by the multi-layered 
planes using the texture mapping method. In figures 
for explaining the third embodiment, the same 
35 reference numerals are assigned to components having 
the same function. 

Figs. 43-51 are schematic diagrams for 
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10 



explaining the principal of the image generation 
method of this embodiment. Figs. 43 and 44 are 
diagrams showing a setting example of the projection 
planes and the reference viewpoint. Fig. 45 is a 
diagram for explaining a method for determining the 
color information and the focusing degree of the 
projection point. Figs. 46-48 are diagrams for 
explaining a method of determining the existence 
probability of the projection point. Fig. 49 is a 
diagram for explaining a method for generating an 
image viewed from the virtual viewpoint. Fig. 50 is 
a diagram for explaining problems in the image 
generation method in this embodiment. Fig. 51 is a 
diagram for explaining a method for solving the 
15 problems in the image generation method in this 

embodiment . 

In the image generation method of this 
invention, as mentioned before, based on plural 
images (multi-focus image) taken by changing 

20 focusing distance from one viewpoint, the three- 
dimensional shape of the object appearing in the 
images is obtained so that an image of the object 
viewed from an arbitrary viewpoint (virtual 
viewpoint) is generated. In this method, the three- 

25 dimensional shape of the object is represented by 
multi-layered planes using the texture mapping 
method. 

For representing the three-dimensional 
shape of the object using the texture mapping method, 

30 the viewpoint C of the camera, projection planes 

Lj (i«l,2,...,M) of the multi-layered structure and the 
reference viewpoint R for obtaining the three- 
dimensional shape of the object are set in a virtual 
three-dimensional space set in the image generation 

35 apparatus such as a computer as shown in Fig. 43, for 
example. Assuming that the shape of the object is 
obtained from N images of different focusing 
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distances, the projection planes Lj are set at each 
distance the same as the focusing distance f± of 
each image Img ± (i=l, 2,„.,N) as shown in Fig. 44. 

As shown in Fig- 44, considering projection 
5 points Tj (j=*l,2, ..„N) overlapping when viewed from 
the reference viewpoint R in a direction, the 
surface of the object is considered to exist at a 
point in the projection points Tj according to a 
conventional model obtaining method. The projection 
10 point, among the projection points Tj, at which the 
surface of the object exists is determined based on 
the focusing degree of each of the projection points 
T jf for example. Then, first, color information Kj 
and focusing degrees Qj of each of the projection 
15 points Tj overlapping viewed from the reference 
viewpoint R are determined. 

The color information Kj and focusing 
degrees Qj of the projection points Tj are determined 
based on the color information Ki of the point 
20 (corresponding point) G±, on each image Imgi, 

corresponding to the projection point Tj, and the 
degree (focusing degree) by which focus is achieved 
at the corresponding point Gi . The color 
information Kj of the projection point Tj is 
25 determined to be an average value of the color 

information K ± of the corresponding points G if or 
determined to be color information K i=j that 
spatially matches with the corresponding point Gi-j - 
The focusing degree of the projection point Tj can 
30 be determined according to sharpness or blurriness 
at a point or a small region on the image- As a 
calculation method for the focusing degree, there 
are various methods based on Depth from Focus theory 
or Depth from Defocus theory. Following documents 
35 can be referred to as for the Depth from Focus 
theory and the Depth from Defocus theory. 

Document 8: A. P . Pentland: "A New Sense for Depth 
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of Field," IEEE Trans* On Pattern Analysis and 
Machine Intelligence, Vol.PAMl-9, No. 4, pp. 523- 
531 (1987) . 

Document 9: :Murali Subbarao and gopal Surya: 
5 "Depth from Defocus: A Spatial Domain Approach," 
International Journal of Computer Vision, 13, 3, 
pp. 271-294, Kluwer Academic Publishers. 

Document 10: Mitsuhiro Ishihara, Hiromi 
Sasaki, : "High speed three-dimensional shape 
10 measurement by Focusing method", Precision 

Engineering, Vol.63, No.l, pp. 124-128, The Japan 
Society for Precision Engineering. 

Document 11 , Koutarou Ooba, Shigeru Yamada: -Real 
time all focusing microscope camera", O plus E, 
15 Vol.22, No. 12, pp. 1568-1576, 2000, Shin Gijyutu 
communications . 

The focusing degree Qj can be obtained by 
comparing sizes of local spatial frequency of each 
corresponding point G±, for example . 
20 The Depth from Focus theory or the Depth 

from Defocus theory is a method for analyzing plural 
images having different focusing distance so as to 
measure the surface shape of the object. In the 
method, the surface of the object is estimated to 
25 exist at a distance corresponding to a focusing 
distance of an image having the highest local 
spatial frequency among the images taken by changing 
the focusing distance, for example. Thus, the 
focusing degree Qj of the projection point Tj is 
30 calculated using an evaluation function of the local 
spatial frequency represented by the following 
equation 65, for example. 
[Equation 65] 
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wherein f indicates a gray scale of a pixel, D 
indicates a constant for normalization that is a 
number of all pixels, (-Lc, -Lr)-(Lc, Lr) and (xi, 
yi)-(xf, yf) respectively indicate small regions for 
5 performing distribution evaluation and smoothing. 

This processing is performed for all 
projection points Tj overlapping when viewed from 
the reference viewpoint R, Then, as shown in Fig. 46, 
after the color information and the focusing degree 
10 Qj are determined for each projection point T 3 , a 

distance at which the surface of the object exists 
is estimated based on the focusing degree Qj of each 
projection point T 5 . As to the focusing degree Qj 
of each of projection points Tj overlapping when 
15 viewed from the reference viewpoint R, when only a 

focusing degree Q n of a projection point T n is a very 
high value as shown in Fig.47A r for example, it can 
be estimated that the surface of the object exists 
on the projection point T n , and the reliability is 
20 very high. 

However, there is a case where there is no 
projection point T at which the focusing degree Q 
takes a distinctive large value depending on the 
condition for taking the images, the shape of the 

25 object or the texture of the surface of the object 
as shown in Fig.47B, for example. In the example 
shown in Fig.47B, for example, since the focusing 
degrees Q of the projection points T n and T n are 
high to some extent compared to other focusing 

30 degrees Q of other projection points T, it can be 

considered that the surface of the object exists at 
either one of the projection points T n and T n > 
However, since any of the focusing degrees Q of the 
projection points T n and T n * is not a distinctive 

35 large value, the reliability is low when one of the 
projection points is selected. In addition, 
according to circumstances, an erroneous projection 
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point may be selected. Then, if erroneous 
estimation (selection) is performed for estimating 
the projection point at which the surface of the 
object exists, a large noise appears on the 
5 generated image . 

In such a case, for increasing the 
reliability of estimation of the distance of the 
surface of the object, that is, for increasing the 
reliability of estimation of a projection plane on 
10 which the surface of the object exists, some 

contrivance to estimate the distance of the object 
surface is necessary in which more images are used 
by decreasing the intervals of the focusing 
distances, or not only the peak position but also 
15 values before and after the peak position are 
applied to a known function such as normal 
distribution function, for example. 

However, when more images are used, there 
are problems that processing time increases, and 
20 that it becomes difficult to adjust the focusing 

distances in the image taking means. In addition, 
when focusing at a focusing distance, there is a 
region called depth of field before and after the 
focus. If a point is within the limit of the depth 
25 of field, blurriness does not substantially occur on 
the image even when the point is not at the focusing 
distance. Therefore, effects can be obtained by 
dividing the focusing distances into smaller 
intervals as long as the interval is decreased up to 
30 about the depth of field. Even if the interval is 

further decreased, the effect is small. In addition, 
when the spatial frequency of the texture of the 
object surface is low, that is, when change of the 
texture is small so that the texture is even, 
35 changes are hard to occur on the image even when the 
focusing distance is changed. In this case, it is 
difficult to estimate with high reliability in 
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principle. 

Therefore, generally, even when the 
distribution of the focusing degree Q is as shown in 
Fig.47B, it is assumed that the surface of the 
5 object exists at the projection point Tj where the 
focusing degree Q is maximum. Therefore, there are 
many cases where erroneous estimation for the 
distance of the object surface is performed so that 
a large noise appears on the generated image. 

10 Thus, according to the image generation 

method of this invention, the distance of the 
surface of the object is not specified as one point, 
that is, the distance of the surface of the object 
is not specified as one of projection points Tj 

15 overlapping when viewed from the reference viewpoint 
R, but existence probability pj is provided 
according to the focusing degree Qj of each 
projection point Tj . In this method, the existence 
probability pj needs to satisfy the following 

20 equation 66 and the equation 67 in a set of the 

existence probability p 3 of all projection points Tj 
overlapping when viewed from the reference viewpoint 
R. 

[Equation 66] 

0</?,<l 

[Equation 67] 
M 

Then, assuming that the number of the projection 
planes Lj is M and that the number of the projection 
30 points Tj overlapping when viewed from the reference 
viewpoint R is M, the existence probability (3 k of 
the projection point T k on a k-th projection plane L k 
is obtained by the following equation 68* 
[Equation 68] 
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Accordingly, by performing processing for 
determining the existence probability (3j of the 
projection point Tj for every projection point Tj 
5 overlapping when viewed from the reference viewpoint 
R for every direction, the three-dimensional shape 
of the object can be obtained. When generating the 
image of the object viewed from the virtual 
viewpoint P, the virtual viewpoint P is set on a 

10 space in which the projection planes Lj are set so 

as to determine the color information of each point 
of the image to be generated as shown in Fig. 49, for 
example. In this method, the color information K A 
of a point A on the image to be generated is 

15 determined by using the following equation 69, for 
example, from color information Kj and existence 
probability pj of projection points Tj overlapping 
with the point A when viewed from the virtual 
viewpoint P. 

20 [Equation 69] 

M 

By determining the color information for every point 
on the image to be generated; the image (virtual 
viewpoint image) of the object viewed from the 

25 virtual viewpoint P is obtained. When only a 

focusing degree Q n of a projection point T n has a 
distinctive large value on the generated virtual 
viewpoint image as shown in Fig.47A, only an 
existence probability p n of the projection point T n 

30 becomes large. Therefore, when performing mixing 
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processing on the color information using the 
equation 69, contribution ratio of the color 
information K n of the projection point T n to the 
color information K A of the point A on the generated 
5 image becomes high so that it is rendered clearly, 
in addition, as shown in Fig.47B, when it is 
difficult to estimate the distance of the surface of 
the object, since the contribution ratio to the 
color information K A of the point A on the generated 
10 image becomes low so that it is rendered vaguely. 
As a result, the discontinuous large noise that 
appears when erroneous estimation of distance is 
performed can be decreased, so that an image that 
looks good for the observer can be generated. 
15 m addition, the image generation method 

of the present invention can be implemented simply 
by using texture mapping that is a basic method in 
computer graphics. Therefore, computing workload 
can be decreased such that processing can be 
20 properly performed by three-dimensional graphics 
hardware mounted on a general personal computer - 

However, in the image generation method of 
this invention, the focusing degree Qj is calculated 
for each of the projection points Tj overlapping 
25 viewed from a viewpoint such as the reference 
viewpoint, and the existence probability is 
determined. Therefore, there is a case where the 
projection points overlapping viewed from the 
virtual viewpoint P include more than one point at 
30 which the existence probability is very high 
depending on the position relationship of the 
reference viewpoint and the virtual viewpoint. In 
such a case, if the color information of each 
projection point are mixed in a ratio according to 
35 the existence probability, there is a case where 
color information of a point on the image to be 
generated may exceed a limit of an effective color 
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information. 

Thus r in the same way as described in 
embodiments described so far, a case is considered 
in which two projection planes Li and L 2 , a 
5 reference viewpoint R and a virtual viewpoint R are 
set in a space where the object exists as shown in 
Fig- 50. In this example, assuming that color 
information of the projection points T if T 2 , T'j. and 
T T 2 are K ir K 2 , K f i and K' 2 r and existence 

10 probabilities of the object are Pi, p 2 , P'i and p f 2 
respectively- 

When the existence probabilities of the 
objectPi, p 2 , P'i and P' 2 are determined on a line 
passing through the reference viewpoint R, the 

15 surface of the object Obj exists near the projection 
points T'i and T 2 in the example shown in Fig ♦ 50- 
Therefore, the existence probability of each of the 
projection points T'i and T 2 becomes higher than 
projection points Ti and T' 2 - In this case, the 

20 existence probabilities Pi, P2, P'i ^ nd P f 2 of the 

projection points Ti, T 2 , T'i and T ' 2 are represented 
by the following equation 70 and equation 71, 
[Equation 70] 

25 [Equation 71] 

A'sl, #=° 

The color information K A at the point A of 
the image plane of the virtual viewpoint P is 
obtained by adding, after assigning weights 

30 according to the existence probabilities PS and p 2 , 
color information K'i and K 2 of the projection points 
T'i and T 2 overlapping with the point A on the image 
plane when viewed from the virtual viewpoint P 
according to the equation 69, and the color 

35 information K A is represented as shown in the 
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following equation 72. 
[Equation 72] 

K A = fiK[+p 2 K 2 

In addition, the equation can be approximated to the 
5 following equation 73 according to the equation 70 
and the equation 71. 
[Equation 73] 

k a 45 K[ + K 2 

However, when the object Obj is viewed from 

10 the virtual viewpoint P, since the projection point 
T' i is blocked by the projection point T 2 , actual 
color information at the point A on the image 
surface should be K A =K 2 - That is, brightness of 
each component (R,G,B) of the generated color 

15 information K A at the point A on the image is 

increased by K 1 1 compared with the actual color 
information . 

In addition, when each component of the 
color information K'i and K 2 of the projection points 

20 T'i and T 2 has large brightness, the color 

information K A of the point A exceeds a limit, of an 
effective color space. Therefore, clipping 
processing is necessary for the color information to 
fall within the limit of the effective color space. 

25 Thus, in the image generation method of 

this invention, transparency that may take plural 
gradations from transparent to opaque is provided to 
each projection point based on the existence 
probability of the projection point. In this case, 

30 the mixing processing is performed successively from 
a projection point of the generated image far from 
the viewpoint to a projection point near the 
viewpoint, in which color information obtained by 
mixing processing up to a projection point is 

35 obtained by performing interior division between 
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color information at the projection point and color 
information obtained by mixing processing up to a 
previous projection point with a ratio according to 
the transparency. The color information obtained by 
5 the mixing processing is interior division between 
color information at a stage and color information 
at the next stage. 

For explaining the principle of the mixing 
processing of the color information, assuming a case 

10 in which projection planes Lj { j=l , 2 , M) , projection 
points Tj, and a vector Kj having components of red 
(R) , green <G) and blue (B) and representing color 
information of the projection point are set in a 
color space V as shown in Fig.51A, for example. It 

15 is assumed that the color information V is 
represented by the following equation 74. 
[Equation 74 ] 

Kj eV,V = {(R,G 9 B)\0<R<\,0<G<l,0<B<l} 

In addition, it is assumed that the transparency ofj 
20 of the projection point Tj is set such that it 
satisfies a condition shown in the equation 75. 
[Equation 75] 

0 < a j < 1 

At this time, the color information D ro obtained by 
25 mixing processing up to the argument j=m is 
represented by recurrence formulas shown in 
equations 76 and 77. Then, color information D M of 
the projection plane L M nearest to the virtual 
viewpoint P, that is, color information D M obtained 
30 by performing mixing processing up to the argument 
j=M becomes the color information K A at the point A 
on the image plane of the image to be generated, 
[Equation 76] 

D m =a m K m +(\-a m )D m _ l 
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[Equation 77] 

D x =a l K l 

In addition, in this case, as shown by the 
relationship of the equations 75 and 76, the color 
5 information D m is an internally divided point 
between the vector K m and the color information 
in the color space V. Therefore, as to the color 
information D mf when K m GV and D m -i^V are satisfied, 
D ro ev holds true as shown in Fig . 51B . 
10 Then, when the conditions of the equations 

74 and 75 are satisfied, it is ensured that the 
following equation 78 holds true with respect to the 
color information D M at the virtual viewpoint P. 
[Equation 78] 

That is, by setting the color information K-j and the 
transparency c*j at the projection point Tj so as to 
satisfy the equations 74 and 75, the color 
information D M of the point h of the image to be 
20 generated can always fall within a proper color 
space V. 

As mentioned above, for generating images 
obtained by viewing the same object from plural 
virtual viewpoints, by performing the mixing 
25 processing of the color information using the 

transparency otj, all images can be generated within 
a limit of proper color information even when 
calculating the color information and the 
transparency of the projection point from a 
30 viewpoint (reference viewpoint) if the color 

information and the transparency satisfies the 
equations 74 and 75. 

Thus, transparencies oti, <*2r a r i and a' 2 
than can be provided by the following equations 79 
35 and 80 are provided to the projection points Ti, T2, 
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10 



T'i and T' 2 respectively, for example, in the example 
shown in Fig. 50. 
(Equation 79] 

[Equation 80] 

For obtaining color information of each point of the 
virtual viewpoint, mixing processing is performed 
successively from a projection point far from the 
virtual viewpoint toward a projection point near the 
projection point, and color information obtained by 
mixing processing up to a projection point is 
obtained by performing interior division between 
color information at the projection point and color 
15 information obtained by mixing processing up to a 
previous projection point at a ratio of 
transparencies, so that color information K A of the 
point A of the image viewed from the virtual 
viewpoint P is represented by the following equation 

20 81- 

[Equation 81) 

K A =a 2 K 2 +{l-a 2 )a[K{ 

The equation 81 can be represented as the following 
equation 82 according to the equation 70, the 
25 equation 71, the equation 79 and the equation 80. 
The equation 82 represents good approximation of 
actual color information. 
[Equation 82] 

As mentioned above, when performing image generation 
using the existence probability information as it is, 
there is no problem if the reference viewpoint P 
and the viewpoint P of the image to be generated are 
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the same. But, when they are different, brightness 
may increase near blocked region. On the other hand, 
according to image generation in which the existence 
probability information is changed to transparency, 
5 the above mentioned phenomenon can be prevented. 

In addition, when the reference viewpoint 
R and the virtual viewpoint P are different in the 
image generation using the existence probability 
information as it is, there is no guarantee that 
10 color information that is calculated using after- 
mentioned equations falls within an effective color 
information limit, and correction processing is 
necessary, for example. On the other hand, 
according to the image generation in which the 
15 existence probability information is converted to 

the transparency, such correction is not necessary. 

In addition, according to the image 
generation in which the existence probability 
information (3 is converted to transparency cc, a 
20 semitransparent object can be efficiently 

represented, so that there is an effect that the 
present invention can be applied more widely to more 
objects in the actual world- 
Next, an example of a mathematical model 
25 used in processing in generating the image using the 
image generation method of the present invention is 
describe . 

Figs. 52 and 53 are schematic diagrams for 
explaining the mathematical model of the image 

30 generation method of the present invention. Fig. 52 
is a diagram, showing relationship among projection 
points, corresponding points and points on the image 
to be generated. Fig. 53 is a diagram for explaining 
a method for converting points on the space to 

35 pixels on the image. 

For generating the image viewed from the 
virtual viewpoint P using the image generation 
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method of this invention, color information or 
brightness information of a point on the image 
viewed from the virtual viewpoint using perspective 
projection conversion, for example. In this example, 
5 a case is considered where the viewpoint C of the 
camera, the projection planes L 3 ( j=l, 2, „.,M) and the 
virtual viewpoint P are set as shown in Fig. 52, for 
example . 

Generally, a matrix for projecting a 
10 projection point T a (X,Y,Z) on the three-dimensional 
space to a point (x,y) of an image viewed from the 
virtual viewpoint P, that is, an image to be 
generated is provided as a 3 rows and 4 columns 
matrix. The projection matrix, and the matrix 0»o 
15 representing the perspective projection conversion 

of focal length f in which the original point is the 
center are the same as those described in the first 
embodiment and the like. 

In addition, also relationship between the 
20 image coordinates (x,y) and the digital image 

coordinates (u,v) shown in Fig. 53 is the same as 
that described in the first embodiment. Although 
the digital image coordinates (u, v) take discrete 
values when writing into or reading out the two- 
25 dimensional array, it is assumed that the 

coordinates take continuous values unless otherwise 
specified, and that proper discretization processing 
is performed when accessing the array. In addition 
to the coordinate conversion, conversion for 
30 correcting distortion by aberration of the lens can 
be performed. 

(Embodiment 3-1) 

Figs. 54-57 are schematic diagrams for 
explaining the image generation method of the 
35 embodiment 3-1 of this invention. Fig. 54 is a flow 

diagram showing a generation procedure for the image. 
Fig. 55 is a diagram for explaining a method for 
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a 



is 



a 



setting the projection point series. Fig. 56 is 
flow diagram showing a concrete example of 
processing of the step 10305 of Fig. 54. Fig. 57 
diagram for explaining a rendering method. 
5 The image generation method of this 

embodiment 3-1 is a method for generating an image 
using the principle described before. As shown in 
Fig. 54, the method includes a step 101 of obtaining 
plural images having different focusing distances, 
10 step 102 of setting a viewpoint (virtual viewpoint) 
of the observer, a step 103 of obtaining the three- 
dimensional shape of the object based on the 
obtained images, and a step 104 of rendering an 
image of the three-dimensional shape of the object, 
15 obtained in the step 103, when viewed from the 
virtual viewpoint. 

The step 103 includes a step 10301 of 
setting projection planes of multi-layered structure, 
a step 10302 of determining the reference viewpoint 
20 for obtaining the three-dimensional shape of the 
object, a step 10303 for setting projection point 
series, corresponding points and the like, a step 

10304 of keeping the texture array, that is, an area 
for storing the color information and the existence 

25 probability of the projection points, and a step 

10305 of determining the color information and the 
existence probability of the projection point. 

In the image generation method of this 
embodiment 3-1, the plural images are obtained by 
30 taking the object by changing the focusing distances 
as shown in Fig. 54, for example (step 101). 
Although the image to be obtained may be a color 
image or a black and white image, it is assumed that 
the color image in which each point (pixel) is 
35 represented by color information using three primary 
colors of red (R>, green (G) and blue (B) is 
obtained in this embodiment 3-1 in the following 
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10 



15 



descriptions . 

Next, a position (virtual viewpoint) at 
which the observer observes the object is set (step 
102) - Next, the three-dimensional shape of the 
object is obtained using the obtained images of the 
object (step 103). Then, after obtaining the three- 
dimensional shape of the object, the image of the 
object viewed from the virtual viewpoint is 
generated (step 104) . 

In the step 103, first, the projection 
planes L a ( j e j, J S { 1 , 2 , .„, M} ) of the multi-layered 
structure are set (step 10301) as shown in Fig. 54, 
for example. As the projection planes L 3 , 
projection planes having plane shape are placed in 
parallel as shown in Fig. 43, for example. It is 
desirable that the placement intervals of the 
projection planes are the same as focusing distances 
of the images obtained in the step 101 as shown in 
Fig. 44, for example, but they may not necessarily be 
20 the same as the focusing distances. 

Next, a viewpoint for obtaining the three- 
dimensional shape of the object is determined, in 
other words, a point (reference viewpoint) R that is 
a reference when obtaining the probability that the 
25 surface of the object exists on the projection point 
is determined (step 10302) . The reference viewpoint 
R may be the same as the virtual viewpoint P or may 
not be the same. When generating images of the 
object viewed from plural virtual viewpoints P 
successively, the reference viewpoint may be 
determined as a barycenter of the virtual viewpoints. 

Next, projection point series that are 
groups each including projection points on a line 
that pass through the reference viewpoint R, and 
35 points (corresponding points) corresponding to the 
projection points are set (step 10303). The 
projection point series are defines as sets of cross 
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points (projection points) T a between the lines that 
pass through the reference viewpoint R and the 
projection planes L 4 . The projection point series S 
can be described as s={T 3 ljej}, and if the set is o, 

5 sSo holds true. 

Next, an array (texture array) for holding 
the image to be texture-mapped to each projection 
point is kept on a memory in an apparatus for 
generating the image, for example (step 10304). At 
10 this time, the array to be kept includes color 

information and existence probability information by 
8 bits for each pixel as texture information 
corresponding to the position of the projection 
point . 

15 In the ste p 10304, correspondence between 

the two-dimensional digital coordinates (U^Vj) of 
the pixel of the texture array and the three- 
dimensional coordinates (X^Y^Zj) of the projection 
point Tg is also set. At this time, for example, 
20 values of (X jr Y Jf Z 3 ) for every (U 3 ,Vj) may be set as 
a table. Or, only some values of (Xj,Yj,Zj) of 
representative (U jf V 3 ) may be set and other 
correspondences are obtained by interpolation 
processing such as linear interpolation, for example. 
25 Next, color information Kj and existence 

probability (Jj for pixels corresponding to each 
projection point Tj kept in the step 10304 are 
determined based on the images of the object 
obtained in the step 101 (step 10305) . At that time, 
30 as to the color information and the existence 
probability, the projection points T 5 on the 
projection point series S are scanned successively 
within a limit of TjSS, and the processing is 
repeated within a limit of the projection point 
35 series S(e a ), so that double loop processing is 
performed. 

In the processing of the step 10305, first, 
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as shown in Fig. 56, the projection point series S to 
be scanned is initialized to a start position (step 
10305a). y Then, next, the projection point T 3 for 
scan is initialized to a start position within the 
5 projection point series S, for example, initialized 

to j=l (step 10305b) . 

Next, color information Kj of coordinates 
(Xj*,Yj\ Zj*) of the projection point T 3 is determined 
(step 10305c) . In the step 10305c, first, a 
10 position, in the image plane (image sensor) , ^ that 
corresponds to a point at coordinates (Xj ,Yj ,Z 3 ) 
when the point is taken is calculated. Then, the 
color information of a pixel (U^Vj*) , on the 
texture array, corresponding to the projection point 
15 Tj is determined to be color information at 
(uij'iVij'KiSl), for example. 

Next, the focusing degree Qj of the 
projection point Tj is determined. The focusing 
degree Qj is calculated using the equation 65 based 
20 on the local spatial frequency of the corresponding 
point, for example (step 10305d) . 

Next, after processing up to the step 
10305d has been performed, the projection point Tj 
is updated, and it is determined whether every 
25 projection point Tj£S has been scanned (step 

10305e) . When every point has been scanned, the 
step goes to the next step 10305f. If every point 
has not been scanned, the step returns to the step 
10305c. 

30 When it is determined that every point 

has been scanned in step 10305e, next, the 
probability (existence probability) Pj that the 
object exists on the projection point is determined 
for every projection point Tj(jej) on the 
35 projection pint series S based on the focusing 

degree Qj of each projection point T 3 calculated in 
the step 10305d (step 10305f ) . The existence 
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probability (5j is determined using the equation 68 , 
for example. Since it is only necessary for the 
existence probability pj to satisfy the equation 66 
and the equation 67 basically, equations other than 
5 the equation 68 can be used. 

Next, the projection point series S is 
updated and it is determined whether every 
projection point series S^o has been scanned (step 
10305g) . If every projection point series has been 
10 scanned, the processing of the step 103, that is, 

processing for obtaining the three-dimensional shape 
of the object completes. If every projection point 
series has not been scanned, the step returns to the 
step 10305b. 

15 in the step 10305g, when it is determined 

that every projection point series has been scanned, 
next, an image of the object, represented by using 
the projection planes Lj ( j=l, 2,„., M) , viewed from the 
virtual viewpoint P is generated by rendering 
20 according to the existence probability (step 104) • 
It is assumed that the coordinates of the image 
plane of the virtual viewpoint P are represented by 
Up and v p as shown in Fig. 57. In this case, the 
color information K p * of a pixel p*(u p *,v p *) on the 
25 image plane is determined by multiplying color 

information {Kj*ljSj} of projection point series 
{Tj*|j^J} on a line connecting between the virtual 
viewpoint P and the pixel p* by corresponding 
existence probability ip/lj^J}, and by adding the 
30 multiplied color information, so that the color 
information K p w is represented by the following 
equation 83 > 
[Equation 83] 

k ;=± p ;k; 
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Then, by determining the color information for every 
pixel on the image plane by using the equation 83, 
the image of the virtual viewpoint P can be obtained. 
By calculating K p * using the following 
5 equation 84 instead of the equation 83, it can be 
ensured that K p * always falls within a limit of an 
effective color space even when positions of the 
reference viewpoint R and the virtual viewpoint P 
are different - 
10 [Equation 84] 

IX*/ 

A /> - M 

Although color information is determined by scanning 
pixels on the image plane in this procedure, the 
procedure is not limited to this. For example, 
15 rendering processing can be performed by a general 

graphic library such as OpenGL, DirectX and the like 
by passing, to the library, data such as the 
configuration of the projection planes L jf the 
texture array, setting of the virtual viewpoint P 

20 and the like. 

Accordingly, the generation processing of 
the virtual viewpoint image of this embodiment 3-1 
completes, so that the generated image is output to 
the image display means such as a CRT (Cathode Ray 

25 Tube), a liquid crystal display and the like. As to 
the image displayed on the image display means, in 
images of the object, color information of a 
projection point having low focusing degree Q 
calculated in the step 10305d has small contribution 

30 to color information of a point on the generated 
image. Therefore, the projection point at which 
reliability of distance estimation is low is 
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rendered vaguely. As a result, for example, the 
image does not appear to lack partial image or to 
have extremely deteriorated image quality as the 
image generated by the conventional method, but the 
5 deterioration is not annoying for the user- 
Figs. 58-61 are schematic diagrams showing 
a general configuration of the apparatus for 
generating the image using the image generation 
method of this embodiment 3-1- Fig. 58 is a block 
10 diagram showing the configuration of the apparatus. 
Figs. 59-61 are diagrams showing a configuration 
example of the object image taking means. 

In Fig. 58 , 2 indicates the image 
generation apparatus, 201 indicates a multiple focus 
15 image (object image) obtaining means, 202 indicates 
a virtual viewpoint setting means, 203 indicates a 
projection plane etc. setting means, 204 indicates a 
texture array keeping means, 205 indicates a color 
information/existence probability determination 
20 means, 206 indicates a rendering means, 207 
indicates a generated image output means, 3 
indicates a multiple focus image (object image) 
taking means, 4 indicates a viewpoint information 
input means, 5 indicates an image display means. In 
25 Fig- 59, 6 indicates a polarizing binary optical 
system, 7, 7A and 7B indicates image sensors, 8 
indicates a beam splitter, ObjA and ObjB indicate 
objects. In Fig. 60, 9 indicates a polarizing filter. 
In Fig. 61, 10 indicates a variable focus lens, 11a, 
30 lib, 11c and lid indicate fixed focus lenses, 12 
indicates a lens holder. 

As shown in Fig. 58, for example, the image 
generation apparatus 2 used for generating the image 
using the image generation method of this embodiment 
35 includes the multiple focus image (object image) 

obtaining means 201 for obtaining plural images of 
different focusing distance, the virtual viewpoint 
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setting means 202 for setting the viewpoint (virtual 
viewpoint) of the image to be generated, the 
projection plane etc , setting means 203 for setting 
projection planes of the multi-layered structure on 
5 the virtual three-dimensional space, the texture 

array keeping means 204 for assigning/ to a memory, 
the array of the image (texture image) to be mapped 
to the projection plane, the color 
information/existence probability determination 
10 means 205 for determining the color information and 
the existence probability on the point (to be 
referred to as projection point hereinafter) on each 
projection plane, the rendering means 206 for 
determining color information of each pixel on the 
15 image to be generated, and the generated image 

output means for outputting the image generated by 
the rendering means 206. 

The object image obtaining means 201 
obtains the image of the object taken by the object 
20 image taking means 3 including a lens, that changes 
focusing distance according to polarization 
component, such as the polarizing binary optical 
system (refer to document 12: Japanese Laid-Open 
Patent Application No. 2000-258738, for example) . 
25 In addition, it is not limited to the polarizing 
binary optical system, and the image taken by an 
image taking means including a variable focus lens 
(refer to document 13: Japanese Patent No- 3303275, 
for example) can be obtained. In addition to that, 
30 by holding lenses, having different focal length, as 
one, images taken by changing lenses at high speed 
can be obtained. In this case, the object image 
obtaining means 201 can successively obtain the 
position and posture of the object that changes 
35 every moment at constant intervals of 30 Hz, for 

example, or can obtain a still image of the object 
at a desired time* In addition, an image recorded 
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in a recording medium that is electrical/ optical or 
magnetic after being taken by the object image 
taking means 3 can be obtained. It is desirable 
that the images of the object are taken at the same 
5 time. However, this is not always so when change of 
the position and the posture of the object is slow 
enough so that the object is regarded as a still 
ob j ect . 

The virtual viewpoint setting means 202 
10 sets position,, direction, image angle and the like, 
for example, as parameters of the viewpoint (virtual 
viewpoint) of the image to be generated. The 
virtual viewpoint may be automatically determined by 
the virtual viewpoint setting means 202, or may be 
15 determined based on information input, by the user, 
using the viewpoint information input means 4 such 
as the mouse or the key board and the like. The 
viewpoint information input means 4 may be a 
position/posture detection sensor mounted by the 
20 user. In addition, the viewpoint information input 
means 4 may be one provided by other program or one 
provided via the network. 

The projection plane etc. setting means 

203 performs processing of the steps 10301, 10302 
25 and 10303 shown in Fig. 54, for example. 

The texture array keeping means 204 
performs the processing of the step 10304 shown in 
Fig. 54, and holds information on the color 
information and the existence probability for each 
30 pixel, for example. The texture array keeping means 

204 holds a texture array representing each of 
three-primary color of red (R> , green (G) and blue 
(B) and the existence probability using 8 bits. 
However, the present invention does not depend on 

35 such particular data representation format. 

The color information/existence 
probability determination means 205 performs 
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processing of the step 10305 shown in Fig. 24, and 
performs processing from the step 10305a to the step 
10305g shown in Fig. 56, for example. The rendering 
means 206 performs processing of the step 104 shown 
5 in Fig. 54 based on the result of the color 

information/existence probability determination 
means 205 so as to generate the image of the object 
viewed from the virtual viewpoint P. 

The virtual viewpoint image generated by 
10 the rendering means 206 is output from the generated 
image output means 207 , and is displayed by the 
image display means 5 such as a CRT, a LCD {Liquid 
Crystal Display) and a PDP (Plasma Display Panel) and 
the like. The image display means 5 may be a 
15 display means like a two-dimensional plane, or may 
be a display means having a curved surface 
surrounding the user. By using a display apparatus 
that can display stereoscopically as the image 
display means 5, images that are independent for 
20 right and left eyes of the user can be presented 
after determining two viewpoints corresponding to 
the right and left eyes of the user by the virtual 
viewpoint setting means 202 and generating a 
stereoscopic image from the two viewpoints. In 
25 addition, by using a three-dimensional display that 
can generate imagse from more than two virtual 
viewpoints and can display images having more than 
two parallaxes, a stereoscopic image can be 
presented to equal to or more than one user. 
30 The generated image output means 207 may 

be a means that outputs the generated image not only 
to the image display means 5 but also to a recording 
medium that is electrical, magnetic or optical, for 
example . 

35 in addition, although not shown in the 

figure, the image generation apparatus 2 can be 
provided with a storing means for storing the 
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generated image to store the generated image, and 
the stored image can be output according to an 
instruction of the user so as to display the image 
on the image display means 5. 
5 in addition, by using a camera in which 

the paralyzing binary optical system is embedded as 
the taking means 3, the object Obj can be taken 
using two kinds of focusing distances. The 
paralyzing binary optical system is an optical 
10 system using material presenting optical anisotropy 
(birefringence). As shown in Fig.59A, for example, 
focusing distances f i and f 2 are different between a 
case where the polarizing component of light passing 
through the paralyzing binary optical system 6 is p 
15 component and a case where the polarizing component 
is s component. When the image is formed on one 
image sensor 7 like a general taking means, as to 
the image obtained from one image sensor 7, an image 
of p component and an image of s component are 
20 overlapped, that is, an image taken with focusing 
distance f i and an image taken with focusing 
distance f 2 are overlapped. Therefore, as shown in 
Fig.59B, for example, the light passed through the 
polarizing binary optical system 6 is separated by a 
25 beam splitter 8 so that the light of the p component 
is focused on a first image sensor 7A and the light 
of the s component is focused on a second image 
sensor 7B. As a result, the image of focusing 
distance fi and the image of focusing distance f 2 can 
30 be separated and each of them can be obtained. 

For example, as shown in Fig.59B, when the 
object ObjA exists near the focusing distance fi and 
the object ObjB exists near the focusing distance f 2 , 
the object ObjA becomes sharp and the object ObjB 
35 becomes blurred in the image of the first image 

sensor 7 A, that is, in the image formed by the light 
of the p component as shown in Fig.59C. In contrast, 
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the object ObjA becomes blurred and the object ObjB 
becomes sharp in the image of the second image 
sensor 7B. 

In addition, for dividing the image taken 
5 by using the polarizing binary optical system 6, a 
polarizing filter 9 may be provided between the 
polarizing binary optical system 6 and the image 
sensor 7 as shown in Fig.60A, for example, instead 
of using the beam splitter 8. In this case, the 
10 placed polarizing filter 9 is configured such that a 
filter 9A for the p component and a filter 9B for 
the s component are arranged like a checked lattice 
as shown in Fig.60B, for example. In this case, 
when the size of each of the filters 9A and 9B is 
15 the same as the size of the pixels of the image 
sensor 7, or is n*n pixels, two images shown in 
Fig.59C can be obtained by subtracting pixels 
corresponding to p component or s component from the 
image obtained by the image sensor 7. 
20 In addition, for taking the plural images 

having different focusing distances, a variable 
focus lens 10 may be used as shown in Fig.61A, for 
example, instead of using the polarizing binary 
optical system 6. By using the variable focus lens 
25 10, images of four focus points fi, f2, fs and f« can 
be obtained using one lens as shown in Fig-61A, for 
example . 

In addition, instead of changing a 
refractive index of a medium of a lens to change the 

30 focus position like the variable focus lens 10 and 
instead of using the polarizing binary optical 
system 6, the images can be taken by holding fixed 
focus lenses 11a, lib, 11c and lid, as one with a 
lens holder 12, each having a fixed focusing 

35 distance that is different each other and by 

rotating the lens holder 12 to switch each lens at 
high speed as shown in Fig.61B. 
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As described above, according to the image 
generation method of this embodiment 3-1, the method 
of the present invention does not intend to obtain a 
geometrical object model accurate for every case and 
5 every portions like the conventional method. But, 
assuming that an estimated value having adequate 
reliability cannot be obtained in the distance 
estimation depending on picture-taking conditions 
and portions, a portion, by which an estimated value 
10 of low reliability is obtained, is drawn vaguely so 
as to provide the portion with low contribution to 
image generation for avoiding extreme image 
deterioration, and a portion, by which distance data 
of high reliability is obtained, is drawn clearly so 
15 as to provide the portion with high contribution to 
image generation- Therefore, deterioration of image 
of the portion having low estimation reliability can 
be inconspicuous so that the virtual viewpoint image 
having small deterioration can be obtained in the 
20 eyes of the user. 

In addition, according to the image 
generation method of this embodiment 3-1, since the 
virtual viewpoint image can be generated using 
texture mapping, workload of the apparatus in the 
25 image generation processing can be decreased and the 
virtual viewpoint image can be generated at high 
speed . 

In addition, the virtual viewpoint image 
generation apparatus 2 does not need to be a 

30 specific apparatus. For example, the virtual 
viewpoint image generation apparatus 2 can be 
realized by a computer a program, for example. In 
such a case, by generating a program that can cause 
the computer to execute each step shown in Figs. 54 

35 and 55 and by executing the program on the computer, 
the virtual viewpoint image having small image 
deterioration can be easily generated at high speed 
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even though the computer is a general personal 
computer. In addition, the program can be provided 
by recording it into a recording medium that is 
magnetic, electrical or optical, and also can be 
5 provided via a network. 

The configuration of the image generation 
apparatus, the method for generating the image and 
processing procedures described in this embodiment 
3-1 are merely examples, and the principal object of 
10 the present invention is to provide the existence 
probability to the image to be mapped to the 
projection planes configured by multi-layers and to 
render a part, at which the reliability of distance 
estimation is low, vaguely by texture-mapping the 
15 part to plural projection planes. Thus, within a 
limit that does not largely differ from the scope, 
the present invention is not dependent on a 
particular processing method or implementation. 

In addition, in the image generation 
20 method of this embodiment 3-1, although a case is 
described as an example in which a color image in 
which the point (pixel) is represented by color 
information using three primary color of red (R) , 
green (G) and blue (B) is obtained and the three- 
25 dimensional shape of the object is generated, the 
image to be obtained is not limited to the color 
image. In the image generation method of this 
embodiment 3-1, a black and white image in which the 
point (pixel) is represented by brightness (Y) and 
30 color-difference (U,v) may be obtained so as to 

generate the virtual viewpoint image. In the case 
when the image to be obtained is the black and white 
image, the virtual viewpoint image can be obtained 
by the procedure described in this embodiment 3-1 
35 using the brightness information (Y) as information 
corresponding to the color information. 

Figs. 62 and 63 are schematic diagrams 
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showing a general configuration of the image 
generation system using the image generation 
apparatus of this embodiment 3-1. Fig .62 is a 
diagram showing a configuration example of the image 
5 generation system, and Fiog.63 is a diagram showing 
another configuration example of the image 
generation system. 

The image generation apparatus 1 of this 
embodiment 3-1 can be applied to the image 
10 generation system shown in Fig. 62, for example. In 
this case, when the user User specifies desired 
viewpoint position, direction and image angle using 
the viewpoint information input means 4 such as a 
mouse, the image generation apparatus 2 obtains the 
15 images of the object Obj taken by the taking means 3. 
The image generation apparatus 1 that obtains the 
images of the object Obj generates the image of the 
object Obj viewed from the viewpoint position, 
direction and image angle specified by the user User 
20 according to the procedure described in this 
embodiment 3-1. Then, the generated image is 
displayed on the image display means 5 so that the 
image is presented to the user User. 

The taking means 3 may be placed at a 
25 place geographically near the place where the user 
User exists, or may be placed in a place 
geographically apart. 

In addition, the image generation 
apparatus 1 of this embodiment 3-1 can be applied 
30 not only to the unidirectional image generation 

system in which relationship between the user User 
and the object Obj is fixed/ as shown in Fig. 62, for 
example, but also to a bi-directional communication 
system such as TV phone o TV conference. 
35 when applying the image generation 

apparatus 2 of this embodiment 3-1 to the bi- 
directional communication system, it is only 
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necessary to provide a taking means 3A for taking 
the user UserA and a taking means 3B for taking the 
user UserB as shown in Fig. 63/ for example. 
Accordingly, the user UserA can generate the image 
5 of the user UserB viewed from a desired viewpoint 
using the image of the user UserB taken by the 
taking means 3B so as to display the image on the 
image display means 4A. In the same way, the user 
UserB can generate the image of the user UserA 

10 viewed from a desired viewpoint using the image of 

the user UserA taken by the taking means 3A so as to 
display the image on the image display means 4B. In 
this example, the image generation apparatus 2 may 
be placd at each of the users UserA and UserB, or 

15 may be placed at either one of the users UserA and 

UserB. In addition, by placing the image generation 
apparatus 2C on the network 13 such as the Internet 
or the company LAN and the like, the image viewed 
from the virtual viewpoint can be generated and 

20 displayed without placing the image generation 

apparatus 2 at any of the users UserA and UserB. 

Although communications between two users 
are shown in Fig. 63, among more users, similar image 
generation can be performed. In the case, assuming 

25 a virtual space used for communication separately 

from the actual space in which users actually exist, 
by providing an image of other user according to 
position relationship with each other, the system 
can provide users with feeling as if to share 

30 virtual space (cyber space) on the network. 

The configuration of the system shown in 
Figs, 62 and 63 shows one application example of the 
image generation apparatus in this invention, and 
the present invention is not necessarily limited to 
35 such configuration. That is, placement, form, 

implementation and the like of each apparatus and 
means can be set as one chooses as long as without 
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exceeding the scope of the present invention. 
(Embodiment 3-2) 

Fig. 64 is a flow diagram showing 
characteristic feature of the embodiment 3-2. In 
5 the embodiment 3-2, in the processing for generating 
the virtual viewpoint image described in the 
embodiment 3-1, image generation is performed by 
converting the existence probability to transparency 
instead of using the existence probability of the 
10 projection point determined in the step 10305f. 

In this case, as to the configuration of 
the image generation apparatus 1 and the whole 
processing procedure, forms similar to those 
described in the embodiment 3-1 can be adopted. 
15 Therefore, only different points are described in 
the following. 

In the embodiment 3-1, color information 
of each point on the image viewed from the virtual 
viewpoint P is determined using the equation 69 by 
20 using the existence probability P 3 determined in the 
step 10305f so as to generate the virtual viewpoint 
image. In that case, as described with Fig. 50, the 
obtained color information may be largely different 
from actual color information of the object surface 
25 according to the shape of the object or the position 
relationship between the reference viewpoint and the 
virtual viewpoint. Thus, in this embodiment 3-2, as 
a method for solving the problem, a method is 
described in which the existence probability is 
30 converted to transparency so as to mix color 

information of each projection point in a ratio 
according to the transparency. In this case, the 
step of converting the existence probability into 
the transparency can be performed after the step 
35 10305f in the step 103, or can be performed between 
the steps 103 and 104. In this embodiment, as shown 
in Fig. 64, the step 105 of determining the 
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transparency by converting the existence probability 
is added just after the step 10305f of determining 
the existence probability. 

In this case, although the texture array 
5 is kept for keeping the color information and the 
existence probability in the step 10304 of keeping 
the texture array in the embodiment 3-1, a texture 
array for holding the color information and the 
transparency is kept in the step 10304 in this 

10 embodiment 3-2. 

The transparency cx 3 is calculated based on 
the existence probability (Jj. Then, like the step 
10305f in the embodiment 3-1, the existence 
probability is calculated once in the step 10305f, 

15 and the transparency otj is calculated in the next 
step 105 in this embodiment 3-2. 

In addition, in step 104 for performing 
rendering processing in this embodiment 3-2, Dj is 
successively calculated according to the equations 

20 11-13 instead of the equation 83 or 84 described in 
the embodiment 3-1. Therefore, color information 
K-j* of a pixel p*(u p ",v p *) on the image plane is 
calculated by the following equation 85. 
[Equation 85] 

= D M 



25 



30 



+ {l-cc M \l-a M _ l )-'(l-a 2 )a l K l 

The image generation method in this embodiment 3-2 
is as mentioned above. The method for calculating 
the transparency a-j based on the existence 
probability p 3 is the same as the method described 
with reference to Fig.l9B in the first embodiment. 

According to this embodiment 3-2, like the 
embodiment 3-1, the virtual viewpoint image in which 
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the partial image deterioration is inconspicuous can 
be generated easily at high speed. 

In addition as described in the 
embodiment 3-1, in image generation using the 
5 existence probability information as it is, there 

may be a case in which brightness increases near the 
blocking area of the object when the reference 
viewpoint and the virtual viewpoint are different ♦ 
On the other hand, as described in the embodiment 3- 
10 2, in the image generation for converting the 

existence probability information to transparency, 
there is an effect to prevent the phenomenon. 
Therefore, a virtual viewpoint image having small 
image deterioration and more similar to the actual 
15 object can be obtained. 

In addition, as described in the 
embodiment 3-1, in the image generation using the 
existence probability information as it is, there is 
no guarantee that mixed color information falls 
20 within an effective color information limit when the 
reference viewpoint and the virtual viewpoint are 
different, so that correction processing is required, 
for example. On the other hand, according to the 
image generation for converting the existence 
25 probability information to transparency like the 
embodiment 3-2, such correction is not necessary. 
Therefore, the image generation processing can be 

simplified. 

In addition, in image generation for 

30 converting the existence probability information to 
the transparency like the virtual viewpoint image 
generation method of this embodiment 3-2, a 
semitransparent object can be efficiently 
represented, so that there is an effect that the 

35 present invention can be applied to more objects in 

the actual world. 

The virtual viewpoint image generation 
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method described in this embodiment 3-2 is an 
example, and the principal object of the present 
embodiment is to generate the virtual viewpoint 
image by converting the existence probability 
5 information to the transparency. Therefore, the 
present invention is not dependent on particular 
calculation method or processing procedure within a 
limit largely exceeds the object. 

Also in the image generation method of 
10 this embodiment 3-2, the image to be obtained may be 
a color image or black and white image. When the 
black and white image is obtained, the mixing 
processing described in this embodiment 3-2 is 
performed using brightness information (Y) as 
15 information corresponding to the color information. 

Fig. 65 is a schematic diagram for 
explaining other generation method in the image 
generation method in this invention- 

In the image generation method in the 
20 embodiment 3-1 and the embodiment 3-2, it is assumed 
that a general lens is used for obtaining the plural 
images having different focusing distances, and 
approximation based on a pinhole optical system is 
performed for projection of the color information. 
25 However, when the images having different focusing 
distances are used like the image generation method 
of this invention, a parallel projecting system can 
be set as shown in Fig. 65 for performing projection 
of the color information by obtaining the plural 
30 images using a telecentric lens. In this case, 
assuming that a cross point of a line, passing 
through the point A on the image to be generated, 
perpendicular to the image surface and the 
projection plane Lj is the corresponding point T 3 , 
35 the probability that the object exists on the 
projection point T 3 is obtained by the procedure 
described in the embodiment. In this case, as shown 
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in Fig- 65, the x component and the y component of 
the corresponding point d to be texture-mapped to 
the projection point Ti become the x component and 
the y component of the point G on the image sensor. 
5 On a generated image Imgi, a part the same as the 

part appearing on the corresponding point G x is on a 
line, passing through the point G on the image 
sensor, perpendicular to the image sensor. Thus, 
regarding these points as corresponding points Gi, 
10 the color information K x and the focusing degree Q x 
of the projection point Ti are determined- Then, 
after performing this processing for each projection 
point T 3 on the same line, the existence probability 
Pj can be obtained from the focusing degree of each 
15 projection point- 

(Effects of the third embodiment) 

As mentioned above, the image generation 
method of the third embodiment provides color 
information or brightness information and the 
20 probability (existence probability) that the surface 
of the object exists to plural projection points 
that overlap when viewed from the reference 
viewpoint for obtaining the three-dimensional shape 
of the object. That is, the method of this 
25 embodiment does not assume that the surface of the 

object exists on a projection point among the plural 
projection points overlapping viewed from the 
reference viewpoint like conventional methods for 
obtaining the three-dimensional shape, but assumes 
30 that the surface of the object exists on each 

projection point with the existence probability. By 
doing so, even when the reliability in distance 
estimation is low, the surface of the object exists 
with a probability on the projection point at which 
35 the surface of the object actually exists. In this 
case, when the image of the object viewed from the 
virtual viewpoint is generated, a pixel, among 
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points (pixels) of the generated image, having color 
information or brightness information obtained by 
mixing color information of projection point at 
which the existence probability is low is rendered 
5 vaguely f that is, a point on the object at which the 
reliability in distance estimation is low is 
rendered vaguely. As a result, discontinuous noise 
that may occur when erroneous distance estimation is 
performed can be made inconspicuous. 

10 In this method, plural images having 

different focusing degree are used for obtaining the 
three-dimensional shape of the object* The 
existence probability of each of the projection 
points overlapping when viewed from the reference 

15 viewpoint is determined based on the focusing degree 
of points (corresponding points) on each image 
corresponding to the projection points. Therefore, 
there may be a case where more than one projection 
points at which the existence probability is very 

20 high are included in the plural projection points 
overlapping when viewed from the virtual viewpoint 
depending on the shape of the object or the position 
relationship between the reference viewpoint and the 
virtual viewpoint- In such a case, for example, 

25 there is a fear that color information on a point of 
the generated image exceeds a limit of an effective 
color space when color information or brightness 
information of each projection point are mixed in a 
ratio according to the existence probability, 

30 Therefore, the transparency may be set at the 

projection point based on the existence probability 
so that the mixing processing of the color 
information can be performed based on a ratio 
according to the transparency- By doing so, when 

35 more than one projection points at which the 

existence probability is high are included in the 
plural projection points overlapping when viewed 
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from the virtual viewpoint, contribution of color 
information of a projection point far from the 
virtual viewpoint to color information of a point on 
the generated image can be decreased, wherein the 
projection point far from the virtual viewpoint is a 
projection point that is not visible from the 
virtual viewpoint, for example. 

This method does not perform processing 
for obtaining an accurate geometrical model for 
every shape of the object and for every point of the 
object as performed in conventional generation 
methods. Therefore, workload on the apparatus 
(computer) that generates the image can be decreased. 
By decreasing the workload on the apparatus that 
15 generates the image, the image can be generated at 
high speed even by an apparatus having low 
processing performance like a generally used 

personal computer. 

In addition, when obtaining the three- 
20 dimensional shape of the object, by using the images 
taken from one viewpoint by changing the focusing 
distance, the taking apparatus used for taking the 
images can be downsized compared to a conventional 
apparatus that takes the images from plural 
25 viewpoints, so that the apparatus configuration can 
be simplified. In this case, for example, by taking 
the object using the polarizing binary optical 
system including a material by which the focusing 
distance changes according to the polarization 
30 planes, two images having different focusing 

distances can be taken from one viewpoint. In 
addition, for example, by providing plural lenses of 
difference focusing distances so as to take the 
images while changing each lens at high speed, more 
35 than two images having different focusing distances 
can be taken from one viewpoint. 
[Fourth embodiment] 



20 0 5I12JI16B 22810 9^ 



1TOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 168 



-158- 



10 



15 



Next, the fourth embodiment of the present 
invention is described. The fourth embodiment 
mainly corresponds to claims 30-43. The fourth 
embodiment is characterized in that the existence 
probability is obtained based on statistical 
processing (parameter fitting) for the evaluation 
reference value v 3 . In figures for explaining the 
fourth embodiment, the same signs are assigned to 
components having the same function. 

In the three-dimensional image display 
method of the fourth embodiment, the three- 
dimensional shape of the object is obtained from 
plural images obtained by taking the object under 
different taking conditions each other. Then, based 
on the three-dimensional shape of the object, a 
three-dimensional image of the object is displayed 
on a display having plural display planes like the 
DFD. When obtaining the three-dimensional shape of 
the object, projection planes of the multi-layered 
structure are set in a virtual three-dimensional 
space, and for points (projection points), on plural 
projection planes, that overlaps when viewed from 
the observer's viewpoint, color information or 
brightness information of each projection point, and 
25 probability (existence probability) that the surface 
of the object exists on the projection point is 
determined. Then, when generating two-dimensional 
images to be displayed on the plural display planes 
based on the three-dimensional shape of the object, 
the color information or brightness information and 
the existence probability are assigned to each point 
of the two-dimensional images. When displaying the 
image on the image display plane, each point of the 
two-dimensional image is displayed with brightness 
35 according to the existence probability. By doing so, 
a part at which the reliability of estimation on 
distance of the surface of the object is low is 



20 



30 



2005#12f]16B 22mi0^ 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 169 



-159- 



displayed vaguely, so that a three-dimensional image 
that looks natural for the observer can be provided. 
(Embodiment 4-1) 

Figs. 66-77 are schematic diagrams for 
5 explaining the image generation method of the 

embodiment 4-1. Fig. 66 is a flow diagram showing an 
example of the whole processing procedure. Figs. 67 
and 68 are diagrams showing an example of a method 
for setting the projection planes. Fig. 69 is a 
10 diagram for explaining a method for setting 

projection point series. Fig. 70 is a flow diagram 
showing an example of processing procedure of steps 
for determining the color information and the 
existence probability of the projection point. 
15 Figs. 71-74 are diagrams for explaining a method for 
determining the existence probability. Figs- 75-77 
are diagrams for explaining a method for generating 
the two-dimensional images to be displayed on each 
image display plane. 
20 The image generation method of this 

embodiment 4-1 is a method for generating images to 
be displayed on an image display means, like a DFD, 
for example, including plural image display planes 
overlapping in the depth direction from the 
25 viewpoint of the observer. As shown in Fig. 66, the 
method includes a step 101 of obtaining plural 
images by taking the object from different 
viewpoints, a step 102 of setting a viewpoint 
(reference viewpoint) of the observer who observes 
30 the three-dimensional image, a step of 103 for 

obtaining the three-dimensional shape of the object 
from plural images, a step 104 of generating two- 
dimensional images to be displayed on each image 
display plane based on the three-dimensional shape 
35 of the object obtained in the step 103, and a step 

105 for providing the three-dimensional image of the 
object by displaying the two-dimensional images 
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generated in the step 104 on each image display 
plane. 

For generating images for displaying the 
tree-dimensional image of the object by the DFD, for 
5 example, using the image generation method of this 
embodiment 4-1, images are obtained by taking the 
object from different viewpoints (step 101). In 
this step, the viewpoints from which the images are 
taken may be arranged in a line or may be arranged 
10 like an arc or on any curved line, or may be 

arranged two-dimensionally on a plane or a curved 
surface. In addition, although the obtained image 
may be a color image or a clack and white image, the 
color image in which each pixel on the image is 
15 represented by color information using three-primary 
colors of red (R) , green (G) and blue (B) is 
obtained in this embodiment 4-1. 

Next, after obtaining the images in the 
step 1, the viewpoint of the observer who observes 
20 the object to be displayed on the DFD is set (step 
102) . In this step, as to the viewpoint of the 
observer, relative position relationship between the 
viewpoint of the observer and a reference image 
display plane in the plural image display planes, 
25 such as a distance from the image display surface, 

and direction of line of sight and the like are set. 

Next, after setting the viewpoint of the 
observer in step 2, the three-dimensional shape of 
the object appearing in the images are obtained from 
30 the plural images obtained in the step 1 (step 103). 
In the step 3, first, projection planes 
Lj(j-1,2,_,M) of the multi-layered structure are set 
(step 10301). Then, next, the reference viewpoint R 
for obtained the three-dimensional shape of the 
35 object is set (step 10302). As the projection 

planes L a , plural planes parallel to the xy plane on 
the virtual three-dimensional space are set as shown 
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in Figs. 67 and 68, for example. In addition, each 
projection plane Lj is set at each distance lj from 
z=0 in a minus direction in the three-dimensional 
space as shown in Figs. 67 and 68. The reference 
viewpoint R is a viewpoint for obtaining the three- 
dimensional shape of the object, and can be set as 
any point in the three-dimensional space. Therefore, 
the reference viewpoint R is determined to be the 
viewpoint of the observer set in the step 2. In 
addition, for example, a projection plane Li 
farthest from z=0 is regarded as an image display 
plane existing in the deepest place from the 
observer of the DFD, and the reference viewpoint R 
is set such that the distance from the projection 
15 plane L x becomes the distance Id from the observer' s 
viewpoint to the image display plane deepest in the 
DFD as shown in Fig. 68. 

Next, after setting the projection planes 
Lj and the reference viewpoint R in the step 10301 
and the step 10302, projection points on the 
projection planes, and points (corresponding points) , 
on the obtained images, corresponding to each 
projection point are set (step 10303). In this step, 
as shown in Fig. 69, lines are drawn to plural 
directions from the reference viewpoint R, and the 
projection points are set as each cross point of 
each line and each projection plane L 3 . When 
estimating the distance of the surface of the object, 
since projection points at which the surface exists 
are estimated from the plural projection points Tj 
on the same line, projection points Tj on the same 
line are dealt, as one, as projection point series s. 

The corresponding point is a point Gij, on 
the image plane of each camera, overlapping with the 
35 viewpoint of the camera when viewing the viewpoint 
Ci of the camera from the projection point T 3 , as 
shown in Fig. 67 and 68. As shown in Fig. 67, when 
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15 



setting a two-dimensional coordinate system (xy 
coordinate system) on each image, two-dimensional 
coordinates (x i3 ,y i3 ) of the corresponding point G i3 
corresponding to the projection point T 3 (X 3 ,Y 3 ,Z 3 ) 
can be obtained by projecting the projection point 
Tj on a two-dimensional point of each image plane. 
This projection can be performed by using a 3 rows 
and 4 culumns projection matrix for projecting a 
point (X,Y,Z) in a three-dimensional space to a 
point (x,y) in a two-dimensional plane. The 
relationship between the coordinates (x i3 ,y i3 ) of the 
corresponding point G i3 in the virtual three- 
dimensional space and the digital image coordinates 
(u,v) is as described so far. 

In the the step 10303, the three- 
dimensional space coordinates (X 3 ,Y 3 ,Z 3 ) of the 
projection point T 3 are associated with the digital 
image coordinates (u i3 ,v i3 ) of the corresponding 
point Gi a . The establishment of the association may 
be performed by providing values of (X 3 ,Y 3 ,Z 3 ) in a 
table for every (u i3 ,v i3 >, or by setting values of 
(X 3 ,Y 3 ,Z 3 ) only for representative values of (u i3 ,v i3 ) 
so as to obtain other points by performing 
interpolation processing such as linear 
25 interpolation, for example. 

In addition, although (u,v) take discrete 
values in the digital image coordinate system, it is 
assumed that (u,v) take continuous values unless 
otherwise specified in the following descriptions, 
30 and it is assumed to perform proper discretization 
processing when accessing the two-dimensional array. 

After completing determining the 
projection point series, corresponding points and 
the like in the step 10303, next, the array (texture 
35 array) for storing, information of the projection 

planes L 3 , that is, the image that is to be texture- 
mapped on the projection planes L 3 is kept (step 
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10304) '. At this time, the array to be kept includes 
color information and existence probability for each 
pixel as texture information, corresponding to the 
position of the projection point Tj, by 8 bits, for 
example . 

Next, after keeping the array for storing 
information of the projection planes in the step 
10304, color information and existence probability 
of each projection point Tj is determined (step 
10305). in the step 10305, as shown in Fig. 70, for 
example, double loop processing is performed in 
which processing for determining color information 
and the existence probability of each projection 
point Tj of projection point series is repeated for 
15 every projection point series that are set. Then, 
first, the projection point series is initialized 
(step 10305a) . Next, the projection point Tj on the 
projection point series is initialized to j-1, for 

example (step 10305b) . 

Next, the color information of the 
projection point Tj is determined (step 10305c). In 
the step 10305c, color information Kj of the 
projection point Tj is determined as an average 
value of color information Ki of corresponding 
points Gi set in the step 10303, for example. 

Next, the correlation degree Q 3 of each 
point on the object that is appearing at each 
corresponding point G tj {iei) corresponding to the 
projection point Tj is obtained (step 10305d) - 
Assuming that a vector for representing the color 
information of the projection point T 3 is K jr and 
that a vector repressing the color information of 
the corresponding point G±j is K iJf the correlation 
degree Q, can be obtained by the following equation 

35 86. 

[Equation 86] 
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point T m takes a distinctive small value, generally, 
as shown in Fig.72A. in this case, it can be 
estimated that the surface of the object exists on 
the projection point T m on the subject projection 
point series, and the reliability is high. 

However, there my be a case where there is 
no projection point at which the correlation degree 
has the distinctive small value, as shown in Fig.72B 
when the correlation degrees Q 3 of each projection 
point T 3 on the projection point series are compared 
depending on the shape of the object, the texture of 
the surface, photographing conditions and the like. 
In this case, even if the surface of the object is 
estimated to exist at a projection point, the 
15 reliability is low and there is a case the 

estimation is erroneous. When the estimation is 
erroneous, the effect appears as a large noise on 
the generated image. 

Thus, a probability (existence 
probability) Pj that the surface of the object 
exists on the projection point T 3 on the projection 
point series is determined based on the correlation 
degree Q a of the projection point Tj . Although the 
existence probability Pj can be directly obtained 
from the correlation degree Q 3 , when there is noise 
on the obtained images so that the reliability of 
the correlation degree Qj is low, the existence 
probability p 3 is affected so that its reliability 
is lowered. Thus, according to the image generation 
method of this embodiment 4-1, an evaluation 
reference value v 3 used as a reference value of the 
existence probability 0 3 is obtained (step 10305f ) . 
It is necessary that the evaluation reference value 
v-j satisfies the following equations 87 and 88. 
35 [Equation 87] 
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[Equation 88] 
M 

7=1 

In addition, for defining the evaluation reference 
value Vj such that the higher the probability that 
5 the surface of the object exists at the projection 
point Tj is, the closer to 1 the evaluation 
reference Vj becomes, the evaluation reference value 
Vj (j^j) can be calculated after performing 
conversion processing represented by the following 
10 equations 89 and 90, for example, for the 

correlation degrees Q-j obtained for each projection 
point Tj on the projection point series. 
[Equation 89] 




15 [Equation 90] 




Basically, the evaluation reference value 
Vj only needs to satisfy the equation 87 and the 
equation 88. Therefore, for the conversion 

20 processing, equations other than the equation 89 and 
the equation 90 can be used. 

The evaluation reference value Vj of each 
projection point Tj calculated using the equations 
89 and 90 can be used as the probability (existence 

25 probability) that the surface of the object exists, 
as mentioned before. But, due to influences of 
noise on the obtained image, there is a case where 
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reliability as the existence probability is not 
sufficiently high. Therefore, next, statistical 
processing (parameter fitting) for the evaluation 
reference value Vj is performed by assuming a 
probability distribution model of the object so as 
to obtain a fitting function p(l) as shown in 
Fig>73A, for example (step 10305g) , 

When it is assumed that the probability 
density distribution of the probability that the 
object exists complies with normal distribution 
(Gaussian distribution) for the distance 1, the 
fitting function p(l> of the evaluation reference 
value Vj can be represented as the following 
equation 91. 
15 [Equation 91] 



10 



25 



2a' 



wherein \i is an average value of the existence 
probability distribution, a is dispersion of 
existence probability distribution, and they are 
20 represented as the following equation 92 and 
equation 93 respectively - 
[Equation 92] 

e = £fijh 

7=1 

[Equation 93] 
M 



After obtaining the fitting function p(l), the 
probability p 3 that the object exists at a distance 
lpj of each projection plane LPi, that is, at each 



2005M16B 22*1241 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 178 



-168- 



corresponding point is determined based on the 
function p(l) (step 10305h) . The existence 
probability p 3 is determined using the following 
equation 94, for example. 
5 [Equation 94] 

7 j = t t Pd)di 

wherein lj" and lj + respectively are a lower limit 
and an upper limit of a distance that contributes to 
the projection plane as shown in Fig.73B, and can 
10 be calculated in the following equation 95 and 
equation 96, 
[Equation 95] 

r= tti±L /: = -oo 
h 2 7 

[Equation 96] 

15 v 2 

By performing the processing from the step 10305c to 
the step 10305h, the color information K 3 and the 
existence probability Pj on each projection point Tj 
on the projection point series are determined as 

20 shown in Fig. 74. The values are stored in the area 
kept in the step 10304- 

As mentioned above, when the probability 
density distribution that the object surface exists 
on the projection point series can be assumed, by 

25 obtaining the existence probability pj based on the 
fitting function p(l) of the evaluation reference 
value Vj, the effect of the noise on the taken image 
can be decreased. 

The equation 91 is an example of the 

30 fitting function. The parameter fitting can be also 



2005^1 2^1 68 m\n 



ITOH INTERNATIONAL PATENT OFFICE 



NO. 7708 P. 179 



-169- 



performed using various functions according to shape 
distribution of then object- For example, the 
parameter fitting can be performed using a 
Laplacian distribution function. 
5 After storing the color information Kj and 

the existence probability (Jj of each projection 
point Tj, the projection point series is updated and 
it is checked whether processing from the step 
10305c to the step 10305h has been performed for 
10 every projection point series determined in the step 
10303 (step 103051)- When there is any projection 
point series for which processing from the step 
10305c to the step 10305h has not been performed, 
the step returns to the step 10305b so that the 
15 processing from the step 10305c to the step 10305h 
is repeated. 

Accordingly, after the processing from the 
step 10305c to the step 10305h are performed for 
every projection point series determined in the step 
20 10303, the processing of the step 10305 (step 103) 
completes, so that the three-dimensional shape of 
the object can be obtained. 

By performing the processing of the step 
103, as shown in Fig -74, for example, the texture 
25 array of each projection point Tj ( 2,.-,M) on the 
projection point series stores the color information 
Kj and the existence probability p j - That is, as to 
the three-dimensional shape of the object obtained 
by the three-dimensional image display method of the 
30 present invention, the surface of the object exits 
not only on one projection point on the projection 
point series as is in the conventional method, but 
the surface of the object exits on each projection 
point . 

35 in the image generation method of this 

embodiment 4-1, image of object viewed from the 
observer is generated based on the obtained three- 
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dimensional shape of the object. In this embodiment 
4-1 , the generated images are displayed on each 
image display plane of a display, such as the DFD, 
having plural image display planes- The method for 
5 generating the two-dimensional images is described 
in this embodiment 4-1. in this case, after 
completing the processing of the step 103, as shown 
in Fig. 66, the color information and the existence 
probability of each projection point are converted 
10 to color information and brightness distribution 
coefficient of each point on the two-dimensional 
image generation plane (step 104) - 

When generating the two-dimensional images 
to be displayed on each image display plane in step 
15 104, the viewpoint of the observer, plural two- 
dimensional image generation planes, and the three- 
dimensional shape of the object obtained in the step 
103 are set, first. In this step, the two- 
dimensional image generation planes LD n (n-1 , 2 , N) 
20 are set such that they overlap with each other in 
the depth direction from the viewpoint P of the 
observer as shown in Fig. 75, for example. In 
addition, the distance ld n from the viewpoint P of 
the observer to the two-dimensional image generation 
25 plane LDn is set such that it becomes the distance 

set in the step 102. At this time, if the number of 
projection planes Lj representing the three- 
dimensional shape of the object and the placement 
intervals are the same as the number of the two- 
30 dimensional image generation planes LD n and the 

placement intervals, the three-dimensional shape of 
the object is set such that the projection planes Lj 
becomes the same as the two-dimensional image 
generation planes LD n , as shown in Fig. 75, for 
35 example. In this case, when the two-dimensional 
image generation planes LD n are planes for 
generating the images displayed on the image display 
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planes of the brightness modulation type DFD, it is 
necessary to determine the color information KD n and 
the brightness distribution coefficient Yn for each 
point (display point) A n , on the two-dimensional 
5 image generation planes LD n , that overlap viewed 
from the viewpoint P of the observer. As shown in 
Fig. 75, when the projection planes Lj representing 
the three-dimensional shape of the object are the 
same as the two-dimensional image generation planes 
10 LD n , the color information KD n of each display point 
A n is determined to be the color information Kj of 
the projection point Tj on the projection plane Lj 
that is overlapping the two-dimensional image 
generation plane LD n on which the display point A n 
15 exists. In addition, the brightness distribution 

ratio Yn of the display point A n is determined to be 
the existence probability p 3 of the projection point 
Tj on the projection plane Lj that is overlapping 
the two-dimensional image generation plane LD n . 
20 After the color information KD and the brightness 
distribution coefficient y are determined for each 
display point A on the two-dimensional image 
generation surfaces LDn, the images generated on the 
two-dimensional image generation planes LDn are 
25 output and are displayed on the image display planes 
of the actual DFD (step 105) . 

However, it is not necessary that the 
number and placement intervals of the projection 
planes Lj for representing the three-dimensional 
30 shape of the object are the same as the number and 
placement intervals of the two-dimensional image 
generation planes LD n . Therefore, next, another 
method for generating the two-dimensional images are 
described in a case where the number and placement 
35 intervals of the projection planes L 3 for 

representing the three-dimensional shape of the 
object are not the same as the number and placement 
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intervals of the two-dimensional image generation 

planes LD n - 

In this case, when the distance from a 
projection plane closest to the viewpoint P of the 
5 observer to a projection plane farthest from the 

viewpoint P is almost the same as the distance from 
a two-dimensional image generation plane closest to 
the viewpoint P to a two-dimensional image 
x generation plane farthest from the viewpoint P, the 
10 projection planes L-j representing the three- 
dimensional shape of the object are set such that 
the projection plane Li farthest from the viewpoint 
P of the observer and the two-dimensional image 
generation plane LDi overlap. By doing so, color 
15 information and brightness distribution coefficient 
Y of each display point A on the two-dimensional 
image generation plane LDi farthest from the 
viewpoint P of the observer become the color 
information K and the existence probability p of 
20 each projection point T on the projection plane L x 
farthest from the viewpoint P of the observer. 

Color information KD and brightness 
distribution coefficients y of each display point A 
on the two-dimensional image generation planes LD 
25 each of which does not overlap any projection plane 
are determined in the following method. 

As to the color information KD and 
brightness distribution coefficients y of each 
display point A on the two-dimensional image 
30 generation planes LD each of which does not overlap 
any projection plane, color information K and 
existence probability (J of each of projection points 
T, on the projection planes L, overlapping with the 
display point when viewed from the viewpoint P of 
35 the observer are assigned to a projection point A on 
the two-dimensional image generation plane LD 
closest with respect to the projection plane L. In 
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this case, the color information KD of the display 
point A is determined to be an average value of 
assigned color information K of projection points T, 
or to be color information L of the projection point 
T, on the projection plane L, closest to the two- 
dimensional image generation plane LD. The 
brightness distribution coefficient y is determined 
to be a sum of pieces of assigned existence 
probabilities (3 of each projection point T. In this 
case, assuming that a set of projection planes L 3 to 
which the two-dimensional image generation plane LD n 
is the closest is {Ljlj^Tn}, the brightness 
distribution ratio Yn of the display point A„ on the 
two-dimensional image generation plane LD„ is 
15 provided by the following equation 97 using the 

existence probability of the projection point Tj 
of the projection plane Lj . 
[Equation 97] 



10 



M 

n = HP, 

20 In the following, a case where position 

relationship between the projection planes Lj and 
the two-dimensional image generation surfaces LD n is 
as shown in Fig.77A is considered. Assuming that 
color information Kj and each existence probability 

25 pj of each of projection points T 3 ( j=l, 2, 3, 4, 5) 

overlapping with the display points Ai and A 2 when 
viewed from the viewpoint P of the observer are 
assigned to a display point A on a two-dimensional 
image generation surface that is closest from the 

30 projection plane, color information and existence 

probabilities of the projection points T lf T 2 and T 3 
are assigned to the display point A x . In this case, 
the color information of the display point Ai may be 
an average value of the color information K x , K 2 and 
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K 3 of the projection points T x , T 2 and T 3 , for 
example, or may be the color information K 2 of the 
projection point closest from the viewpoint of 
the display point Ai. The brightness distribution 
5 coefficient Yi of the display point Ai is determined 
to be a sum of the existence probabilities pi, p2 
and p 3 of the projection points T X/ T 2 and T 3 using 
the equation 91. 

In the same way, color information and 
10 existence probabilities of the projection points T 4 
and T 5 are assigned to the two-dimensional image 
generation surface LD^ , In this case, the color 
information KD 2 of the display point A 2 may be an 
average value of the color information K 4 and K 5 of 
15 the projection points T 4 and T 5 , or may be the color 
information K 5 of the projection point T 5 . The 
brightness distribution coefficient Y2 is determined 
to be a sum of the existence probabilities p 4 and p 5 
of the projection points T 4 and T 5 using the equation 
20 91- 

When the placement intervals of the two- 
dimensional image generation planes LD n and the 
placement intervals of the projection planes Lj are 
different, color information and existence 
25 probability of the projection point on a projection 
plane Lj existing between two successive two- 
dimensional image generation planes LD n and LD n+ i can 
be distributed in a ratio of the distances of the 
two two-dimensional image generation planes LD n and 
30 LD n+ i. In this case, assuming that a set of the 
projection planes Lj between the two-dimensional 
image generation planes LD n and LD n+ i is {Ljlj^r n }, 
the brightness distribution ratio Yn of the display 
point A n on the two-dimensional image generation 
35 plane LD n can be calculated by the following 

equation 98 using the existence probability p 3 of 
each projection points Tj - 
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[Equation 98] 

In the equation 98, w 3 ,„ is a coefficient indicating 
a degree of contribution of the projection plane L 3 
5 to the two-dimensional image generation plane LD n . 

A case where projection planes Li and L 2 
are set between the two-dimensional image generation 
planes LDi and LD 2 is considered as shown in Fig.77B, 
for example. In this case, assuming that distances 

10 from the projection plane Li to the display planes 
LD X and LD 2 are Bi and B 2 respectively, the degrees 
wi,j. and w 1#2 of contribution of the projection plane 
Li to the two-dimensional image generation planes LDi 
and LD 2 can be provided by the following equation 99. 

15 [Equation 99] 

In the same way, when distances from the projection 
plane L 2 to the display planes LDi and LD 2 are B 3 and 
B 4 respectively, the degrees w 2(1 and w 2(2 of 
contribution of the projection plane L 2 to the two- 
dimensional image generation planes LDi and LD 2 can 
be provided by the following equation 100. 
(Equation 100] 

As a result, each of the brightness 
distribution ratio Yi of the display point A x of the 
two-dimensional image generation plane LDi and the 
brightness distribution ratio y 2 of the display 
point A 2 of the two-dimensional image generation 
plane LD Z is shown in the following equation 101. 
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[Equation 101] 

ri = w u A. /2 = w i.2A + w 2,2A 

As mentioned above, when obtaining the three- 
dimensional shape of the object, a shape is obtained 
5 in which the probability (existence probability) (Jj 
that the surface of the object exists on each 
projection point Tj is provided from the correlation 
degree Qj of the projection point Tj on the 
projection point series. Then, the brightness 
10 distribution coefficient of the display point A on 
the two-dimensional image generation plane LD is 
provided as the existence probability P j - 
Accordingly, when there is no projection point 
having the correlation degree Qj of a distinctive 
15 value in the projection points Tj on the projection 
point series, so that reliability for estimation of 
the distance of the surface of the object is low, 
the surface of the object is represented vaguely on 
plural projection planes on the projection point 
20 series. Then, the brightness distribution 

coefficient Y of points on the two-dimensional image 
generation plane LD is determined from the existence 
probability p of each projection point Tj . 
Accordingly, when the two-dimensional images 
25 generated on the two-dimensional image generation 
planes are displayed on the actual image display 
planes so as to present the three-dimensional image 
of the object, the surface of the object is 
displayed vaguely on the projection point series in 
30 which reliability for distance estimation is low and 
the existence probability (5 is dispersed to plural 
projection points. Therefore, a noise on the three- 
dimensional image displayed on the DFD becomes 
inconspicuous so that an image that looks natural 
35 for the observer can be displayed. 

As described above, according to the image 
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generation method of this embodiment 4-1, the three- 
dimensional image that looks natural for the 
observer can be displayed without obtaining the 
accurate three-dimensional shape of the object to be 
5 displayed. 

In addition, by obtaining the existence 
probability (1, by assuming probability density 
distribution of the probability that the surface of 
the object exists on the projection point series and 
10 by performing statistical processing for the 

evaluation reference value v 3 calculated from the 
correlation degree Qj, deterioration of the 
reliability of the existence probability fa due to 
noise on the obtained images can be decreased. 
15 m addition, in the image generation 

method of this embodiment 4-1, although a case is 
described by taking an example in which a color 
image in which the point (pixel) is represented by 
color information using three primary color of red 
20 (R) , green (G) and blue (B) is obtained and the 

three-dimensional shape of the object is generated, 
the image to be obtained is not limited to the color 
image. In the image generation method of this 
embodiment 4-1, a black and white image in which the 
25 point (pixel) is represented by brightness (Y) and 

color-difference (U,V) may be obtained to obtain the 
three-dimensional shape of the object. In the case 
when the image to be obtained is the black and white 
image, the three-dimensional shape can be obtained 
30 by the procedure described in this embodiment 4-1 

using the brightness information (Y) as information 
corresponding to the color information so that the 
two-dimensional images can be generated- 
(Embodiment 4-2) 
35 Figs. 78-81 are schematic diagrams for 

explaining the image generation method of the 
embodiment 4-2. Fig. 78 is a diagram showing 
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relationship between the projection point and the 
corresponding point. Fig. 79 is a flow diagram 
showing an example of steps for determining the 
color information and the existence probability of 
5 the projection point. Figs. 80 and 81 are diagrams 
for explaining a method for obtaining the existence 
probability. 

The basic procedure of the image 
generation method of this embodiment 4-2 is the same 
10 as the image generation method of the embodiment 4-1. 
When generating the plural two-dimensional images to 
be displayed on the DFD, processing from the step 
101 to the step 105 shown in Fig- 66 are performed. 
The different point of the image generation method 
15 of this embodiment 4-2 compared to the image 

generation method of the embodiment 4-1 is that 
plural images of different focusing distances are 
obtained instead of the plural images of different 
viewpoints in step 101, and the three-dimensional 
20 shape of the object is obtained using the images 

having different focusing distances in the step 103. 

For generating the plural two-dimensional 
images for displaying the three-dimensional image of 
the object on the DFD using the image generation 
25 method of the embodiment 4-2, plural images are 

taken from a viewpoint while changing the focusing 
distance, first. At this time, the plural images 
are taken using the polarizing binary optical system, 
a variable focus lens and the like, for example. 
30 The image to be obtained may be a color image in the 
same way as the embodiment 4-1 or may be a black and 
white image- Next, as described in the embodiment 
4-1, after setting the viewpoint of the observer 
(step 102), processing of the step 103 for obtaining 
35 the three-dimensional shape of the object is 
performed. 

In the processing of the step 103, as 
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described in the embodiment 4-1, projection planes 
( j =1 , 2 , M) and the reference viewpoint R are set 
first (step 10301 f step 10302). Then, next, the 
projection point series and the corresponding points 
5 are set, and the array (area) for storing the 

information of the projection plane is kept (step 
10303, step 10304) - 

In the three-dimensional image display 
method of this embodiment 4-2 in which the three- 
10- dimensional image of the object is displayed using 
plural images having different focusing distances, 
when setting the projection planes Lj in step 10301, 
the planes are set such that the distances from the 
viewpoint c of the camera are the same as the 
15 focusing distances fi 2, ...,N) of the images taken 

by the camera as shown in Fig. 78, for example. Then, 
in the step 10303, the corresponding points Gi 
corresponding to the projection point T-> is 
determined to be points, on images Img if overlapping 
20 with the projection point Tj when viewing the 

projection point Tj from the viewpoint C of the 
camera. Methods for setting the projection point 
series, and for associating the coordinates of the 
projection point Tj with the digital image 
25 coordinates of the corresponding point G± are the 

same as the methods described in the embodiment 4-1. 
Thus, detailed descriptions are not provided here- 
in addition, the processing for keeping 
the area for storing the information of the 
30 projection planes in the step 10304 is also the same 
as that described in the embodiment 4-1. Thus, 
detailed descriptions are not provided here. 

Next, the color information and the 
existence probability information of each projection 
35 point Tj are determined using the obtained plural 

images (step 10305) . Also in the three-dimensional 
image display method of this embodiment 4-2, in the 
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step 10305, double loop processing is performed in 
which processing for determining the color 
information and the existence probability of each 
projection point Tj on the projection point series 
5 is repeated for every projection point series. Thus, 
first, the projection point series is initialized 
(step 10305a) . Then, next, the projection point Tj 
on the projection point series is initialized to j-l r 
for example (step 10305b) . 
10 Next, the color information of the 

projection point Tj is determined (step 10305c). In 
the step 10305c, an average value of color 
information of the corresponding points Gi set in 
the step 10303 is determined to be the color 
15 information Kj of the projection point T j • 

Next, the focusing degree Qj of the 
projection point Tj is obtained based on degree 
(focusing degree) by which focus is achieved for a 
point, of the object, appearing on each 
20 corresponding point Gi corresponding to the 

projection point Tj (step 10305j). The focusing 
degree of the projection point Tj can be determined 
according to sharpness or blurriness at a point or a 
small region on the image* As a calculation method 
25 for the focusing degree, there are various methods 
based on Depth from Focus theory or Depth from 
Defocus theory. The focusing degree Qj can be 
obtained by comparing sizes of local spatial 
frequency of each corresponding point Gi, for 
30 example. 

The Depth from Focus theory or the Depth 
from Defocus theory is a method for analyzing plural 
images having different focusing distance to measure' 
the surface shape of the object. In the method, the 
35 surface of the object is estimated to exist at a 

distance corresponding to a focusing distance of an 
image having the highest local spatial frequency 
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among the images taken by changing the focusing 
distance, for example. Thus, the focusing degree Qj 
of the projection point Tj is calculated using an 
evaluation function of the local spatial frequency 
represented by the following equation 102, for 
example , 
[Equation 102] 



wherein f indicates a gray scale of a pixel, D 
indicates a constant for normalization that is a 
number of all pixels, (-Lc, -Lr)-(Lc r Lr) and (xi, 
yi)-(xf, yf) indicate small regions for performing 
distribution evaluation and smoothing respectively „ 



method for obtaining the focusing degree Qj, and the 
focusing degree Qj can be obtained by using 
equations other than the equation 102* 



the step 10305j, the projection point Tj is updated, 
and it is checked whether processing of the steps 
10305c and 10305j are performed for every projection 
point on the projection point series that is a 
subject for processing (step 10305e) , When there is 
any projection point for which the processing of the 
steps 10305c and 10305j has not been performed, the 
step returns to the step 10305c so that the color 
information Kj and the focusing degree Qj are 
obtained. 



focusing degree Qj are obtained for every projection 
point on the projection point series that is a 
subject of processing, color information Kj and the 
focusing degree Qj are provided for each projection 
point Tj on the projection point series as shown in 




The equation 102 is one example for a 



After obtaining the focusing degree Qj by 



When the color information and the 
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Fig.80. The focusing degree Qj of the projection 
point Tj is a degree corresponding to the 
correlation degree used for determining the 
existence probability p in the embodiment 4-1. 
5 There my be a case where there is no projection 
point at which the focusing degree has the 
distinctive small value when the focusing degrees Qj 
of each projection point Tj on the projection point 
series are compared depending on the shape of the 

10 object, the texture of the surface, photographing 

conditions and the like. In this case, even if the 
surface of the object is estimated to exist at a 
projection point, the reliability is low and there 
is a case the estimation is erroneous. When the 

15 estimation is erroneous, the effect appears as a 
large noise on the generated image. 

Thus, according to the three-dimensional 
image display method of this invention, next, as 
shown in Fig. 81, the probability (existence 

20 probability) Pj that the surface of the object 

exists on each projection point Tj on the projection 
point series is determinewd. In this case, as 
described in the embodiment 4-1, the existence 
probability pj is determined after performing 

25 statistical processing for the evaluation reference 
value Vj to prevent deterioration of the reliability 
due to the noise of the obtained images (step 
I0305f ) . For calculating the evaluation reference 
value Vj in the step 10305f , it is necessary that 

30 the evaluation reference value Vj satisfies the 

equations 87 and 88. Thus, in this embodiment 4-2, 
the evaluation reference value Vj of the projection 
point Tjt is determined using the equation 103, for 
example . 

35 [Equation 103] 
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7=1 



It is only necessary that the evaluation 
reference value Vj satisfies the conditions of the 
equations 87 and 88. Therefore, the evaluation 
5 reference value \>j can be determined using equations 
other than the equation 97. 

After calculating the evaluation reference 
value Vj in step 10305f r next, parameter fitting is 
performed according to the before-mentioned 

10 procedure, so that the existence probability (3j of 
each projection point Tj is determined as shown in 
Fig. 81 (steps 10305g and 10305h) 

After determining the existence 
probability (3j of each projection point Tj in the 

15 step 10305h, the color information Kj and the 

existence probability pj of each projection point Tj 
are stored in the area kept in the step 10304. 

After storing the color information Kj and 
the existence probability (3 3 of each projection 

20 point Tj, the projection point series is updated and 
it is checked whether processing from the step 
10305c to the step 10305h has been performed for 
every projection point series determined in the step 
10303 (step 10305i) . When there is any projection 

25 point series for which processing from the step 

10305c to the step 10305h has not been performed, 
the step returns to the step 10305b so that the 
processing from the step 10305c to the step 10305h 
is repeated. 

30 Accordingly, after the processing from the 

step 10305c to the step 10305h are performed for 
every projection point series determined in the step 
10303, the processing of the step 10305 completes. 
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so that the three-dimensional shape of the object 
can be obtained. Then, after obtaining the three- 
dimensional shape of the object by the processing of 
the step 103, color information and the brightness 
5 distribution coefficient y of the display point A on 
the two-dimensional image generation plane LD are 
determined based on the three-dimensional shape of 
the object, so as to generate the two-dimensional 
images to be displayed on the plural overlapping 
10 image display planes such as the DFD according to 

the same procedure as the embodiment 4-1 (step 104) . 
Then, by displaying the generated images on actual 
image display planes (step 105) , the three- 
dimensional image of the object can be presented , 
15 Also in the three-dimensional image 

display method of this embodiment 4-2, like the 
three-dimensional image display method of this 
embodiment 4-1, when there is no projection point 
having the focusing degree Qj of a distinctive value 
20 in the projection points Tj on the projection point 
series, so that reliability for estimation of the 
distance of the surface of the object is low, the 
surface of the object is represented vaguely on 
plural projection planes on the projection point 
25 series. Then, the brightness distribution 

coefficient y of points on the two-dimensional image 
generation plane LD is determined from the existence 
probability (3 of each projection point T j . 
Accordingly, when the two-dimensional images 
30 generated on the two-dimensional image generation 
planes are displayed on the actual image display 
planes so as to present the three-dimensional image 
of the object, the surface of the object is 
displayed vaguely on the projection point series in 
35 which reliability for distance estimation is low and 
the existence probability £ is dispersed to plural 
projection points- Therefore, a noise on the three- 
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dimensional image displayed on the DFD becomes 
inconspicuous so that an image that looks natural 
for the observer can be displayed. 

As described above, according to the image 
5 generation method of this embodiment 4-1, the three- 
dimensional image that looks natural for the 
observer can be displayed without obtaining the 
accurate three-dimensional shape of the object to be 
displayed. 

10 In addition, also in the case of the image 

generation method of this embodiment 4-2, the image 
to be obtained may be either of the color image or a 
black and white image- When using the black and 
white image, processing described in this embodiment 

15 4-2 is performed using the brightness information 
(Y) as the information corresponding to the color 
information . 

(Embodiment 4-3) 

Figs. 82-84 are schematic diagrams for 

20 explaining an arbitrary viewpoint image generation 
method of the embodiment 4-3 according to the 
present invention. Fig. 82 is a flow diagram showing 
an example of the whole processing procedure. 
Fig. 83 is a diagram for explaining the principle of 

25 rendering- Figs.84A and 84B are flow diagrams 
showing processing procedure for converting the 
existence probability to the transparency. 

In the embodiments 4-1 and 4-2, although a 
method for generating the two-dimensional images to 

30 be displayed on each image display plane of the 

apparatus having plural image display planes like 
the DFD using the three-dimensional shape of the 
object obtained in the step 103 is taken as an 
example, use of the three-dimensional shape of the 

35 object is not limited to that, but it can be also 
used for generating two-dimensional images of the 
object viewed from arbitrary viewpoints. In this 
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case f the difference compared to the embodiments 4-1 
and 4-2 is that rendering is performed after the 
step 103, that is, processing of step 106 for 
generating the two-dimensional image of the object 
5 viewed from the observer's viewpoint from the three- 
dimensional shape. Since the processing from the 
step 101 to step 103 in which the three-dimensional 
shape of the object is obtained is as described in 
the embodiments 4-1 and 4-2, it is not described in 
10 detail. 

In the step 11 for rendering in the 
arbitrary viewpoint image generation method of this 
embodiment 4-3, the color information of each point 
(pixel) on the arbitrary viewpoint image to be 

15 displayed is determined by performing mixing 

processing for color information Kj of projection 
points Tj ( j=l, 2, ...,M) overlapping with a point A on 
the arbitrary viewpoint image as shown in Fig. 83, 
for example- Mixing processing for color 

20 information using the transparency cxj is performed 
as described in the embodiment 2-2 and the like in 
the second embodiment. 

Also in the method for generating the 
arbitrary viewpoint image of this embodiment 4-3, 

25 after performing the step I0305h for determining the 
existence probability |3j, processing for converting 
the existence probability v 3 to the transparency ctj 
is performed as shown in Fig.84A, for example (step 
107) . 

30 In the processing for converting the 

existence probability Vj to the transparency cx jf the 
projection point Tj is initialized to j=M first as 
shown in Fig.84B, for example (step 107a). Next, £ M 
is substituted to transparency <* M of the projection 

35 point T M such that ot M =(S M (step 107b) . 

Next, the value of the argument j is 
updated to j-j-1 (step 107c) > Then, it is 
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determined whether the transparency a 3 +i is 1 (step 
107d) . When transparency c* 3+ ]>l, the transparency a 3 
is obtained from the equation 104, for example (step 
107e) . 
5 [Equation 104] 

Pi 

n(i-«j 

When the transparency otj+i is 1, 1 is substituted 
into ofj (aj=l) , for example (step 107f ) . When 
obtaining the transparency c<j in the step 107e, the 

10 equation is not limited to the equation 104, and 

other equations can be used. In addition, although 
detailed descriptions are not provided, since any 
value can be substituted into the transparency aj 
actually in step 107e, a value other than 1 may be 

15 substituted into the transparency ctj . 

Next, it is determined whether processing 
from the step 107d to the step 107f has been 
performed up to the argument j=l (step 107g) . When 
the processing has not been performed, the 

20 processing is repeated after returning to the step 
107c. 

After performing the processing from the 
step I07d to the step 107f up to the argument j=l, 
processing for converting the existence probability 

25 Vj of the projection point Tj overlapping with the 
point A on the image surface viewed from the 
observer's viewpoint P to the transparency c<j 
completes. After that, mixing processing using the 
equation 62 and the equation 63 is performed in the 

30 rendering step 104 to obtain the color information 
D M on the point A of the arbitrary viewpoint image . 
Then, by performing the processing for every point 
(pixel) on the arbitrary viewpoint image, the 
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arbitrary viewpoint image from the observer' s 
viewpoint can be obtained. 

In addition, also in the case of the image 
generation method of this embodiment 4-3, the image 
5 to be obtained may be either of the color image or a 
black and white image. When using the black and 
white image, after obtaining the three-dimensional 
shape of the object by performing processing 
described in the embodiment 4-1 using the brightness 

10 information (Y) as the information corresponding to 
the color information, the virtual viewpoint image 
can be generated according to the procedure 
described in this embodiment 4-3. 
(Embodiment 4-4) 

15 Figs. 85-89 are schematic diagrams showing 

general configurations of an image generation 
apparatus this embodiment 4-4 according to the 
present invention. Figs. 85 and 86 are block 
diagrams showing a configuration example of the 

20 image generation apparatus. Figs. 87 and 88 are 

diagrams showing a configuration example of an image 
display system using the image generation apparatus. 

In Figs. 85 and 86, 2 indicates the three- 
dimensional image generation apparatus, 201 

25 indicates an object image obtaining means, 202 

indicates a observer viewpoint setting means, 203 
indicates a projection plane etc. setting means, 204 
indicates a texture array keeping means, 205 
indicates a color information/existence probability 

30 determination means, 206 indicates a projection 
plane information - display plane information 
conversion means, 207 indicates an image output 
means, 208 indicates a rendering means, 3 indicates 
an image display means, 4 indicates an object image 

35 taking means, and 5 indicates an observer viewpoint 
input means. 

The image generation apparatus 2 in this 
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embodiment 4-4 is an apparatus for obtaining the 
three-dimensional shape of the object according to 
the procedure described in the embodiments 4-1 and 
4-2, and generating the two-dimensional images to be 
5 displayed on each image display plane of the image 
display means 3 like the DFD having plural 
overlapping image display planes, and the image of 
the object viewed from arbitrary viewpoints. As 
shown in Fig. 85, for example, the apparatus for 

10 generating images to be displayed on the DFD 

includes the object image obtaining means 201 for 
obtaining plural object images taken under different 
photographing conditions, the observer viewpoint 
setting means 202 for setting a viewpoint of the 

15 observer who observes a generated image, the 

projection plane etc. setting means 203 for setting 
projection planes, projection point series, 
projection points, corresponding points and the like 
for determining the existence probability, the 

20 texture array keeping means 204 for keeping an area 
for storing color information and existence 
probability of points (projection points) on the 
projection plane, the color information/existence 
probability determination means 205 for determining 

25 the color information of the projection points and 
the probability (existence probability) that the 
object exist at the projection points, the 
projection plane information - display plane 
information conversion means 206 for converting the 

30 color information and the existence probability 
information of the projection point to color 
information and existence probability of a point of 
the two-dimensional image e to be displayed on the 
image display plane, and the image output means 207. 

35 An image output from the image output means 207 is 
displayed by the image display means 3 such as the 
DFD including plural overlapping display planes- 
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In addition,, as described in the 
embodiment 4-3, in the apparatus for generating the 
image of the object viewed from the arbitrary 
viewpoint, the rendering means 208 is provided 
5 instead of the projection plane information - 

display plane information conversion means 206 as 
shown in Fig. 86. Although not shown in the figure, 
the apparatus may be configured to include both of 
the projection plane information - display plane 

10 information conversion means 206 and the rendering 
means 208 and to generate a specified image using 
either one of the means according to an instruction 
from the observer. 

The object image obtaining means 201 

15 obtains the images of the object taken by the object 
image taking means 4. The object image taking means 
4 may be a taking means including plural cameras at 
plural viewpoints, or may be a taking means that can 
take images having different focal positions from 

20 one viewpoint- In this case, when the object image 
taking means 4 takes images having different focal 
positions from one viewpoint, the polarizing binary 
optical system (refer to document 12, for example) 
or a variable focus lens (refer to document 13, for 

25 example) can be used. In addition, images can be 

taken while switching plural lenses having different 
focal positions at high speed. 

The observer viewpoint setting means 202 
sets distance and the like between the observer's 

30 viewpoint and the image display plane of the image 

display means 3 based on information input using the 
viewpoint information input means 5 such as the 
mouse or a keyboard, by the observer for example. 
The viewpoint information input means 5 may be a 

35 means for detecting posture or line of sight' of the 
observer so as to input information according to the 
posture or the line of sight. 
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The projection plane etc. setting means 
203 sets parallel projection planes Lj r projection 
point series, corresponding points and the like as 
described in the embodiments 4-1 and 4-2. 
5 The texture array keeping means 204 keeps 

the area for the color information Kj and the 
existence probability Pj of each projection point Tj 
on each projection plane in a memory provided in the 
apparatus, for example, as descried in the 

10 embodiments 4-1 and 4-2* 

The color information/existence 
probability determination means 205 determines the 
color information from the projection point Tj and 
corresponding points Gij on the image, and determines 

15 the probability (Sj that the surface of the object 

exists on the projection point Tj as descried in the 
embodiments 4-1 and 4-2. 

The projection plane information - display 
plane information conversion means 207 converts the 

20 color information and the existence probability of 
the projection plane into color information and the 
brightness distribution ratio of a point (display 
point) on the two-dimensional image to be displayed 
on each image display plane of the image display 

25 means as described in the embodiment 4-1. In the 
case when the rendering means 208 is provided 
instead of the projection plane information - 
display plane information conversion means 206, 
color information of each point on images to be 

30 generated are determined based on the equation 59, 
or the relationship of the equation 62 and the 
equation 63 as described in the embodiment 4-3. 

The image generation apparatus 2 in this 
embodiment 4-4 generates the images to be displayed 

35 on the DFD according to the procedure described in 
the embodiment 4-1 and the embodiment 4-2, for 
example. That is, the three-dimensional image 
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generation apparatus 2 does not need to perform 
processing for obtaining the accurate three- 
dimensional shape of the object as was performed in 
the conventional technology. Therefore, even by an 
5 apparatus that does not have high processing ability 
can generate the images to be displayed on the DFD 
easily and at high speed. 

The image generation apparatus 2 of this 
embodiment 4-4 can be also realized by a computer 

10 and a program executed by the computer , for example. 
In such as case, a program describes instructions 
corresponding to the processing procedure described 
in the embodiment 4-1, or instructions corresponding 
to the processing procedure described in the 

15 embodiment 4-2 is executed on the computer. In 

addition, the program can be provided by a recording 
medium that is magnetic, electrical or optical, or 
can be provided via a network such as the Internet, 
In addition, the image display system 

20 using the image generation apparatus 2 of this 

embodiment 4-4 can be considered to be configured as 
shown in Fig. 87, for example. In the configuration, 
the object image taking means 4 may be placed at a 
place near the space where the observer User 

25 observes the image display means (DFD) 3, or may be 
placed at a place geographically apart. When the 
object image taking means 4 is placed at the 
geographically apart place, the taken image can be 
transferred to the three-dimensional image 

30 generation apparatus 2 using a network such as the 
Internet . 

In addition, as shown in Fig, 87, the image 
display system using the image generation apparatus 
2 of this embodiment 4-4 can be applied not only to 
35 a case where an observer User observes an object Obj 
but also to a bi-directional communication system 
such as TV phone and TV conference ♦ In the case, 
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spaces in which the observers UserA and UserB exist 
are respectively provided with image generation 
apparatuses 2A and 2B, image display means (DFDs) 3A 
and 3B, object image taking means 4A and 4B, and 
5 reference viewpoint setting means 5A and 5B as shown 
in Fig, 88, for example. Then, for example, by 
connecting between the image generation apparatuses 
placed at the spaces where the observers UserA and 
UserB exist by the network 6 such as the Internet, 
10 the observer UserA can observe, by the image display 
means 3A, the three-dimensional image of the 
observer UserB generated from the image taken by the 
object image taking means 4B- In the same way, the 
observer UserB can observe, by the image display 
15 means 3B, the three-dimensional image of the 

observer UserA generated from the image taken by the 
object image taking means 4A. 

In addition, it is not necessary both of 
the image generation apparatuses 2A and 2B are 
20 configured as shown in Fig, 88 for applying to such 
bi-directional communication system, either one of 
the image generation apparatuses 2A and 2B can be a 
general communication terminal that does not include 
the configuration means shown in Fig. 86. In 
25 addition, each configuration means shown in Fig. 86 
may be distributed to the image generation 
apparatuses 2A and 2B. 

In addition, as shown in Fig. 88, if 
another image generation apparatus 2C is provided on 
30 the network 6, the three-dimensional image of the 
object to be displayed on the image display means 
(DFD) 3A and 3B can be obtained using the image 
generation apparatus 2C on the network 6 even when 
the image generation apparatuses 2A and 2B are not 
35 placed at the spaces where the observers UserA and 
UserB exists. 

In addition, in Figs, 87 and 88, although a 
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systera is shown in which the object image taking 
means 4 includes plural cameras, the number of 
cameras may be one for generating the display image 
as shown in Fig,89 f for example, when obtaining the 
5 three-dimensional shape of the object from images 
having different focusing distances as described in 
the embodiment 4^2. 

As mentioned above, although the present 
invention is described concretely based on 

10 embodiments, the present invention is not limited to 
the specifically disclosed embodiments, and 
variations and modifications may be made without 
departing from the scope of the invention. 

For example, although a method for 

15 displaying the three-dimensional image of the object 
based on images of different viewpoints is described 
in the embodiments 4-1, and a method for displaying 
the three-dimensional image of the object based on 
images having different focusing distances is 

20 described in the embodiment 4-2, the three- 
dimensional image of the object can be displayed by 
combining these methods. In this case, the 
correlation degree is obtained from corresponding 
points of images of different viewpoints with 

25 respect to a projection point T jf the local space 
frequency is obtained from corresponding points of 
images taken by changing focal positions from a 
viewpoint, and the existence probability Pj is 
obtained by combining them. By doing so, the 

30 reliability of the existence probability pj 

increases so that an image that looks natural for 
the observer can be displayed. 

(Effects of the fourth embodiment) 

In the image generation method of the 

35 fourth embodiment, when obtaining the three- 
dimensional shape of the object, as mentioned before, 
plural projection planes are set, and the 
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probability (existence probability) that the surface 
of the object exists on the projection point is 
provided to each of the points (projection points), 
on each projection plane, overlapping when viewed 
5 from the reference viewpoint. That is, the method 
of this embodiment does not assume that the surface 
of the object exists on a projection point in the 
plural projection points overlapping viewed from the 
reference viewpoint like conventional methods for 

10 obtaining the three-dimensional shape, but assumes 
that the surface of the object exists on each 
projection point with the existence probability. In 
addition, when determining the existence probability, 
statistical processing is performed on the 

15 evaluation reference value that is calculated from 
the correlation degree or the focusing degree so 
that the existence probability of the object is 
generated from the evaluation reference value on 
which the statistical processing has been performed. 

20 At this time, in the statistical processing, a 
probability distribution model of the object is 
assumed, so that the fitting function of the 
evaluation reference value of each projection point 
is obtained, and the existence probability of each 

25 projection point is determined from that. 

By doing so, when viewing a direction from 
the reference viewpoint, even when the reliability 
for estimating a distance (projection point) at 
which the object surface exists is low, the surface 

30 of the object exists with a probability on the 

projection point at which the surface of the object 
actually exists. As a result, by generating the 
image of the object viewed from the viewpoint of the 
observer based on the three-dimensional shape of the 

35 object obtained according to the procedure, 

discontinuous noise that may occur when erroneous 
distance estimation is performed can be made 
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inconspicuous. In addition, by determining the 
existence probability by performing the statistical 
processing for the evaluation reference value, 
deterioration of the reliability of the existence 
5 probability due to effects of noise of the obtained 
images can be decreased. 

In addition, according to this embodiment, 
even by an apparatus having low processing 
performance such a generally used personal computer, 
10 each two-dimensional image can be generated at high 
speed. 

[ Fifth embodiment] 

Next, the fifth embodiment of the present 

15 invention is described. The fifth embodiment mainly 
corresponds to claims 44-53. In the fifth 
embodiment, the three-dimensional shape of the 
object is obtained from plural images obtained by 
taking the object under different taking conditions. 

20 Then, the three-dimensional image of the object is 

displayed on a display, like a DFD, including plural 
display planes based on the three-dimensional shape 
of the object* In the fifth embodiment, processing 
of the parameter fitting described in the fourth 

25 embodiment is not performed. In figures for 

explaining the fifth embodiment, the same signs are 
assigned to components having the same function. 

When obtaining the three-dimensional shape 
of the object, projection planes of the multi- 

30 layered structure are set in a virtual three- 
dimensional space, and for points (projection 
points), on plural projection planes, that overlap 
when viewed from the observer's viewpoint, color 
information or brightness information of each 

35 projection point, and probability (existence 

probability) that the surface of the object exists 
on the projection point are determined. Then, when 
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generating two-dimensional images to be displayed on 
the plural display planes based on the three- 
dimensional shape of the object, the color 
information or brightness information and the 
5 existence probability are assigned to each point of 
the two-dimensional images* When displaying the 
image on the image display plane, each point of the 
two-dimensional image is displayed with brightness 
according to the existence probability. By doing so f 

10 a part at which the reliability of estimation on 

distance of the surface of the object is displayed 
vaguely, so that a three-dimensional image that 
looks natural for the observer can be provided. 
(Embodiment 5-1) 

15 Figs- 90-100 are schematic diagrams for 

explaining the three-dimensional image generation 
method of the embodiment 5-1. Fig. 90 is a flow 
diagram showing an example of the whole processing 
procedure. Figs. 91 and 92 are diagrams showing an 

20 example of a method for setting the projection plane. 
Fig. 93 is a diagram for explaining a method for 
setting projection point series. Fig. 94 is a flow 
diagram showing an example of processing procedure 
of steps for determining the color information and 

25 the existence probability of the projection point. 
Figs, 95-97 are diagrams for explaining a method for 
determining the existence probability- Figs. 98-100 
are diagrams for explaining a method for generating 
the two-dimensional images to be displayed on each 

30 image display plane „ 

The three-dimensional image generation 
method of this embodiment 5-1 is a method includes, 
as shown in Fig. 90, for example, a step 101 of 
obtaining plural images by taking the object from 

35 different viewpoints, a step 102 of setting a 

viewpoint (reference viewpoint) of the observer who 
observes the three-dimensional image, a step 103 for 
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obtaining the three-dimensional shape of the object 
from plural images, a step 104 of generating two- 
dimensional images to be displayed on each image 
display plane based on the three-dimensional shape 
5 of the object obtained in the step 103, and a step 
105 for providing the three-dimensional image of the 
object by displaying the two-dimensional images 
generated in the step 104 on each image display 
plane, 

10 For displaying the tree-dimensional image 

of the object on the DFD using the image generation 
method of this embodiment 5-1, images are obtained 
by taking the object from different viewpoints (step 

101) - In this step, the viewpoints from which the 
15 images are taken may be arranged in a line or may be 

arranged like an arc or on any curved line, or may 
be arranged two-dimensionally on a plane or a curved 
surface. In addition, although the obtained image 
may be a color image or a black and white image, ot 

20 is assumed that the color image in which each pixel 
on the image is represented by color information 
using three-primary colors of red (R) , green (G) and 
blue (B) is obtained in this embodiment 5-1. 

Next, after obtaining the images in the 

25 step 101, the viewpoint of the observer who observes 
the object to be displayed on the DFD is set (step 

102) . In this step, as to the viewpoint of the 
observer, relative position relationship between the 
viewpoint of the observer and a reference image 

30 display plane in the plural image display planes, 

such as a distance from the image display plane, and 
direction of line of sight and the like are set. 

Next, after setting the viewpoint of the 
observer in step 102, the three-dimensional shape of 

35 the object appearing in the images is obtained from 
the plural images obtained in the step 101 (step 

103) - In the step 3, first, projection planes 
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Lj ( j = l, 2, M) of the multi-layered structure are set 
(step 10301) . Then, next, the reference viewpoint R 
for obtained the three-dimensional shape of the 
object is set (step 10302). As the projection 
5 planes Lj, plural planes parallel to the xy plane on 
the virtual three-dimensional space are set as shown 
in Fig. 91, for example. In addition, each 
projection plane Lj is set at a distance lj from z=0 
in a minus direction in the three-dimensional space. 

10 The reference viewpoint R is a viewpoint for 

obtaining the three-dimensional shape of the object, 
and can be set as any point in the three-dimensional 
space. Therefore, the reference viewpoint R is 
determined to be the viewpoint of the observer set 

15 in the step 102. In addition, for example, a 

projection plane L x farthest from z=0 is regarded as 
an image display plane existing in the deepest place 
from the observer of the DFD, and the reference 
viewpoint R is set such that the distance from the 

20 projection plane L a becomes the distance Id from the 
observer's viewpoint to the image display plane 
deepest in the DFD . 

Next, after setting the projection planes 
Lj and the reference viewpoint R in the step 10301 

25 and the step 10302, projection points on the 

projection planes, and points (corresponding points), 
on the obtained images, corresponding to each 
projection point are set (step 10303) . In this step, 
lines are drawn to plural directions from the 

30 reference viewpoint R, and the projection point is 
set as a cross point of each line and each 
projection plane Lj as shown in Fig- 93- When 
estimating the distance of the surface of the object, 
since a projection point at which the object surface 

35 exists is estimated from the plural projection 

points Tj on the same line, projection points Tj on 
the same line are dealt, as one, as projection point 
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series s. 

The corresponding point is a point G^, on 
the image plane of each camera, overlapping with the 
viewpoint of the camera when viewing the viewpoint 
5 Ci of the camera from the projection point Tj, as 
shown in Figs -91 and 92* As shown in Fig. 91, when 
setting a two-dimensional coordinate system (xy 
coordinate system) on each image plane, two- 
dimensional coordinates (Xi 3 ,yij) of the 
10 corresponding point Gij corresponding to the 

projection point Tj(Xj,Yj,Zj) can be obtained by 
projecting the projection point Tj on a two- 
dimensional point .of each image plane. This 
projection can be performed by using a 3 rows and 4 
15 columns projection matrix for projecting a point 
(X,Y,Z) in a three-dimensional space to a point 
(x,y) in a two-dimensional plane. The relationship 
between the coordinates (xij,yij) of the 
corresponding point Gij in the virtual three- 
20 dimensional space and the digital image coordinates 
(u,v) is as described so far. 

In the step 10303, the coordinates 
(Xj,Yj,Zj) of the projection point Tj are associated 
with the digital image coordinates (Uij,Vij). The 
25 establishment of the association may be performed by 
providing values of (Xj,Yj,Zj) in a table for every 
(Uij,v i:j ), or by setting values of (Xj,Y jf Zj) only for 
representative values of (uij,Vij) so as to obtain 
other points by performing interpolation processing 
30 such as linear interpolation, for example. 

In addition, although (u,v) take discrete 
values in the digital image coordinate system, it is 
assumed that (u,v) take continuous values unless 
otherwise specified in the following descriptions, 
35 and it is assumed to perform proper discretization 

processing when accessing the two-dimensional array. 

After determining the projection point 
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10 



series, corresponding points and the like in the 
step 10303, the array for storing, information of 
the projection plane L 3 , that is, the image that is 
to be texture-mapped on the projection plane L 3 is 
kept (step 10304). At this time, the array to be 
kept includes color information and existence 
probability information for each pixel, by 8 bits 
for example, as texture information corresponding to 
the position of the projection point T 3 . 

Next, after keeping the array for storing 
information of the projection plane in the step 
10304, color information and existence probability 
of each projection point Tj is determined (step 
103O5). In the step 10305, as shown in Fig. 94, for 
15 example, double loop processing is performed in 

which processing for determining color information 
and the existence probability of each projection 
point Tj of projection point series is repeated for 
every projection point series that are set. Then, 
20 first, the projection point series is initialized 

(step 10305a) . Next, the projection point Tj on the 
projection point series is initialized to j=l, for 
example (step 10305b) . 

Next, the color information of the 
25 projection point Tj is determined (step 10305c). In 
the step 10305c, color information Kj of the 
projection point T 3 is determined as an average 
value of color information Ki of corresponding 
points Gi set in the step 10303, for example. 
30 ' Next, the correlation degree Qj of each 

point on the object that is appearing at each 
corresponding point ^(1^1) corresponding to the 
projection point Tj is obtained (step 10305d) . 
Assuming that a vector for representing the color 
35 information of the projection point Tj is Kj, and 
that a vector repressing the color information of 
the corresponding point G xi is K i3 , the correlation 
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degree Q 3 can be obtained by the following equation 
105. 

[Equation 105] 

5 By obtaining the correlation degree Q 3 using the 
equation 105, the correlation degree Q 3 always 
becomes a positive value, and, the higher the 
correlation, the smaller the value is. 

The equation 105 is an example method for 
10 obtaining the correlation degree Qj, and the 
correlation degree Qj can be obtained using 
equations other than the equation 86. In addition, 
for obtaining the correlation degree Qj, not only 
each one point of the projection point Tj and the 
15 corresponding points Gij is considered, but also a 
small region including plural points near the 
projection point Tj and the corresponding point 
can be considered. 

After obtaining the correlation degree Qj 
20 at the step I0305d, the projection point Tj is 

updated, and it is checked whether processing of the 
step 10305c and the step 10305d has been performed 
for every projection point on the projection point 
sexies that is a subject of processing (step 10305e) . 
25 If there is any projection point for which the 

processing of the step 10305c and the step 10305d 
has not been performed, the step returns to the step 
10305c to obtain the color information Kj and the 
correlation degree Qj . 
30 After obtaining the color information and 

the correlation degree Qj for every projection point 
on the projection point series that is a subject of 
processing, the color information Kj and the 
correlation degree Qj are provided to each 
35 projection point Tj on the projection point series 
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as shown in Fig. 95. When correlation degrees Qj of 
the projection points T 3 are compared with each 
other, only a correlation degree Q m of a projection 
point T m takes a distinctive small value, generally, 
5 as shown in Fig.96A. In this case, it can be 

estimated that the surface of the object exists on 
the projection point T B on the subject projection 
point series, and the reliability is high. 

However, as described in previous 
10 embodiments, there may be a case where there is no 

projection point at which the correlation degree has 
the distinctive small value, as shown in Fig.96B 
when the correlation degrees Qj of each projection 
point T, on the projection point series are compared 
15 depending on the shape of the object, the texture of 
the surface, photographing conditions and the like, 
in this case, even if the surface of the object is 
estimated to exist at a projection point, the 
reliability is low and there is a case the 
20 estimation is erroneous. When the estimation is 
erroneous, the effect appears as a large noise on 
the generated image. 

Thus, according to the image generation 
method of the present invention, next, the 
25 probability (existence probability) (J 3 that the 

object surface exists at the projection point T 3 on 
the projection point series is obtained (step 
10305f) . It is necessary that the existence 
probability Pj satisfies the following equations 106 

30 and 107. 

[Equation 106] 

[Equation 107] 
M 

/=1 
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In addition, for defining the existence probability 
p-i such that the higher the probability that the 
surface of the object exists at the projection point 
T-j is, the closer to 1 the existence probability pj 
5 is, the existence probability Pj can be 

calculated after performing conversion processing 
represented by the following equations 108 and 109, 
for example, to the correlation degrees Qj obtained 
for each projection point Tj on the projection point 
10 series. 

[Equation 108] 

~ 1 
[Equation 109] 

/-I 

15 Basically, the existence probability Pj 

only needs to satisfy the equation 106 and the 
equation 107. Therefore, for the conversion 
processing, equations other than the equation 108 
and the equation 109 can be used, 
20 After determining the existence 

probability pj of each projection point Tj by the 
step 10305f , the color information Tj and the 
existence probability Pj of each projection point Tj 
are stored in the area kept in the step 10304* 

25 After storing the color information Kj and 

the existence probability pj of each projection 
point Tj, the projection point series is updated and 
it is checked whether processing from the step 
10305c to the step 10305f has been performed for 

30 every projection point series determined in the step 
10303 (step I0305g) . When there is any projection 
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point series for which processing from the step 
10305c to the step 10305f has not been performed, 
the step returns to the step 10305b so that the 
processing from the step 10305c to the step 10305f 

5 is repeated. 

Accordingly, after the processing from the 
step 10305c to the step 10305f are performed for 
every projection point series determined in the step 
10303, the processing of the step 10305 completes, 
10 so that the three-dimensional shape of the object 
can be obtained. 

By performing the processing of the step 
103, the texture array of each projection point 
T-j ( j=l,2,...,M) on the projection point series stores 
15 the color information Kj and the existence 

probability pj as shown in Fig. 97, for example. 
That is, as to the three-dimensional shape of the 
object obtained by the three-dimensional image 
display method of the present invention, the surface 
20 of the object exits not only on one projection point 
on the projection point series as is in the 
conventional method, but the surface of the object 
exits on each projection point. In the three- 
dimensional image display method in the present 
25 invention, two-dimensional images to be displayed on 
each of the plural image display planes are 
generated using the three-dimensional shape of the 

object (step 104) . 

When generating the two-dimensional images 

30 to be displayed on each image display planes in step 
104, the viewpoint of the observer, plural two- 
dimensional image generation planes, and the three- 
dimensional shape of the object obtained in the step 
103 are set, first. In this step, the two- 

35 dimensional image generation planes LD„ (n=l, 2, N) 
are set such that they overlap with each other in 
the depth direction from the viewpoint P of the 
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observer as shown in Fig. 98, for example. In 
addition, the distance ld„ from the viewpoint P of 
the observer to the two-dimensional image generation 
plane LDn is set such that it becomes the distance 
5 set in the step 102. At this time, if the number of 
projection planes L 5 representing the three- 
dimensional shape of the object and the placement 
intervals are the same as the number of the two- 
dimensional image generation planes LD n and the 
10 placement intervals, the three-dimensional shape of 
the object is set such that the projection planes Lj 
becomes the same as the two-dimensional image 
generation planes LD n . In this case, when the two- 
dimensional image generation planes LD n are planes 
15 for generating the images displayed on the image 

display planes of the brightness modulation type DFD, 
it is necessary to determine the color information 
KD n and the brightness distribution coefficients Yn 
for points (display points) A„, on the two- 
20 dimensional image generation plane LD n , that overlap 
viewed from the viewpoint P of the observer. As 
shown in Fig. 98, when the projection planes Lj 
representing the three-dimensional shape of the 
object are the same as the two-dimensional image 
25 generation planes LD n , the color information KD a of 
each display point A n is determined to be the color 
information Kj of the projection point T 3 on the 
projection plane Lj that is overlapping the two- 
dimensional image generation plane LD n on which the 
30 display point A a exists. In addition, the 

brightness distribution ratio y n of the display 
point A n is determined to be the existence 
probability (Jj of the projection point T 3 on the 
projection plane Lj that is overlapping the two- 
3 5 dimensional image generation plane LD„. After the 

color information KD and the brightness distribution 
coefficient y are determined for each display point 
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A on the two-dimensional image generation planes LD ft , 
the images generated on the two-dimensional image 
generation planes LD n are output and are displayed 
on the image display planes of the actual DFD (step 
5 105) . 

However, it is not necessary that the 
number and placement intervals of the projection 
planes L, for representing the three-dimensional 
shape of the object are the same as the number and 

10 placement intervals of the two-dimensional image 
generation plane LD n . Therefore, next, another 
method for generating the two-dimensional images are 
described in a case where the number and placement 
intervals of the projection planes L 3 for 

15 representing the three-dimensional shape of the 

object are not the same as the number and placement 
intervals of the two-dimensional image generation 
plane LD n . 

In this case, when the distance from a 
20 projection plane closest to the viewpoint P of the 
observer to a projection plane farthest from the 
viewpoint P is almost the same as the distance from 
a two-dimensional image generation plane closest to 
the viewpoint P to a two-dimensional image 
25 generation plane farthest from the viewpoint P, the 
projection planes Lj representing the three- 
dimensional shape of the object are set such that 
the projection plane L t farthest from the viewpoint 
P of the observer and the two-dimensional image 
30 generation plane LDi overlap as shown in Fig. 99, for 
example. By doing so, color information and 
brightness distribution coefficients Y of each 
display point A on the two-dimensional image 
generation plane LDi farthest from the viewpoint P 
35 of the observer become the color information K and 

the existence probability p of each projection point 
T on the projection plane L x farthest from the 
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viewpoint P of the observer. 

Color information and brightness KD 
distribution coefficients y of each display point A 
on the two-dimensional image generation plane LD 
5 which does not overlap any projection plane are 
determined in the following method. 

As to the color information and brightness 
KD distribution coefficients Y of the display point 
A on the two-dimensional image generation plane LD 
10 which does not overlap any projection plane, color 
information K and existence probability (J of the 
projection points T, on projection planes L, 
overlapping with the display point when viewed from 
the viewpoint P of the observer are assigned to the 
15 projection point A on the two-dimensional image 
generation plane LD closest for the projection 
planes L. In this case, the color information KD of 
the display point A is determined to be an average 
value of assigned color information K of projection 
20 points T, or to be color information L of the 

projection point T on the projection plane L closest 
to the two-dimensional image generation surface LD. 
The brightness distribution coefficient y is 
determined to be a sum of pieces of assigned 
25 existence probabilities p of each projection point T 
In this case, assuming that a set of projection 
planes L 3 to which the two-dimensional image 
generation plane LD„ is the closest is {Ljljer„}, 
the brightness distribution ratio y a of the display 
30 point A n on the two-dimensional image generation 
surface LD n is provided by the following equation 
110 using the existence probability (Jj of the 
projection point T 3 of the projection plane L j . 
[Equation 110] 



35 
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In the following, a case where position 
relationship between the projection planes Lj and 
the two-dimensional image generation planes LD n is 
as shown in Fig.lOOA is considered. Assuming that 
5 color information K 3 and each existence probability 
(3-5 of each of projection points Tj (j-l, 2, 3, 4, 5) 
overlapping with the display points Ai and A 2 when 
viewed from the viewpoint P of the observer are 
assigned to a display point A on a two-dimensional 
10 image generation plane that is closest for the 

projection planes, color information and existence 
probabilities of the projection points Ti, T 2 and T 3 
are assigned to the display point A 1 . In this case, 
the color information of the display point A 1 may be 
15 an average value of the color information K lf K 2 and 
K 3 of the projection points Ti, T 2 and T 3 , for 
example, or may be the color information K 2 of the 
projection point T 2 closest from the viewpoint of 
the display point Ai . The brightness distribution 
20 coefficient Y i ^ the display point Ai is determined 
to be a sum of the existence probabilities Pi, p 2 
and p 3 of the projection points Ti, T 2 and T 3 using 
the equation 110. 

In the same way, color information and 
25 existence probabilities of the projection points T 4 
and T 5 are assigned to the two-dimensional image 
generation plane LD 2 . In this case, the color 
information KD 2 of the display point A 2 may be an 
average value of the color information K 4 and K 5 of 
30 the projection points T 4 and T 5 , or may be the color 
information K 5 of the projection point T 5 . The 
brightness distribution coefficient Y 2 is determined 
to be a sum of the existence probabilities p 4 and p 5 
of the projection points T 4 and T 5 using the equation 
35 HO. 

When the placement intervals of the two- 
dimensional image generation plane LD n and the 
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placement intervals of the projection planes Lj are 
different, color information and existence 
probability of the projection point on a projection 
plane Lj existing between two successive two- 
5 dimensional image generation plane LD„ and LD n+ i can 
be distributed in a ratio of the distances of the 
two two-dimensional image generation plane LD„ and 
LD n+i . In this case, assuming that a set of the 
projection planes Lj between the two-dimensional 

10 image generation planes LD n and LD n +i is {Lj|jer n }, 
the brightness distribution ratio Yn of the display 
point A n on the two-dimensional image generation 
plane LD n can be calculated by the following 
equation 111 using the existence probability Pj of 

15 each projection points T j . 
[Equation 111] 

In the equation 111, w 3 , a is a coefficient indicating 
a degree of contribution of the projection plane Lj 

20 to the two-dimensional image generation surface LD n - 
A case where projection planes Li and L 2 
are set between the two-dimensional image generation 
planes LDi and LD2 is considered as shown in Fig.lOOB, 
for example. In this case, assuming that distances 

25 from the projection plane Li to the display planes 
LDi and LD 2 are Bi and B 2 respectively, the degrees 
w 1( i and w lr2 of contribution of the projection plane 
Li to the two-dimensional image generation planes LD X 
and LD2 can be provided by the following equation 

30 112. 

[Equation 112] 
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in the same way, when distances from the projection 
plane L 2 to the two-dimensional image generation 
planes LDi and LD 2 are B 3 and B 4 respectively, the 
degrees w 2 ,i and w 2 , 2 of contribution of the 
5 projection plane L 2 to the two-dimensional image 

generation planes LDi and LD 2 can be provided by the 
following equation 113. 
(Equation 113] 

10 a s a result, each of the brightness 

distribution ratio Yi of the display point Aj. of the 
two-dimensional image generation plane LDj and the 
brightness distribution ratio y 2 of the display 
point A 2 of the two-dimensional image generation 

15 plane LD 2 is shown in the following equation 114. 
[Equation 114] 

? \ = w u A + w 2A p 29 y 2 = w U2 p x + w 2j2 fi 2 

As mentioned above, when obtaining the three- 
dimensional shape of the object, a shape is obtained 
20 in which the probability (existence probability) p d 
that the surface of the object exists on each 
projection point T 3 is provided from the correlation 
degree Q 3 of the projection point Tj on the 
projection point series. Then, the brightness 

25 distribution coefficient of the display point A on 
the two-dimensional image generation plane LD is 
provided as the existence probability 3j • 
Accordingly, when there is no projection point 
having the correlation degree Q 3 of a distinctive 

30 value in the projection points Tj on the projection 
point series, so that reliability for estimation of 
the distance of the surface of the object is low, 
the surface of the object is represented vaguely on 
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plural projection planes on the projection point 
series. Then, the brightness distribution 
coefficient Y of the point on the two-dimensional 
image generation plane LD is determined from the 
5 existence probability (3 of each projection point T 3 . 
Accordingly, when the two-dimensional images 
generated on the two-dimensional image generation 
planes are displayed on the actual image display 
planes so as to present the three-dimensional image 
10 of the object, the surface of the object is 

displayed vaguely on the projection point series in 
which reliability for distance estimation is low and 
the existence probability p is dispersed to plural 
projection points. Therefore, a noise on the three- 
15 dimensional image displayed on the DFD becomes 

inconspicuous so that an image that looks natural 
for the observer can be displayed. 

As described above, according to the image 
generation method of this embodiment 5-1, the three- 
20 dimensional image that looks natural for the 

observer can be displayed without obtaining the 
accurate three-dimensional shape of the object to be 
displayed. 

In addition, in the image generation 
25 method of this embodiment 5-1, although a case is 
described as an example in which a color image in 
which the point (pixel) is represented by color 
information using three primary color of red (R) , 
green (G) and blue (B) is obtained and the three- 
30 dimensional shape of the object is generated, the 
image to be obtained is not limited to the color 
image. In the image generation method of this 
embodiment 5-1, a black and white image in which the 
point (pixel) is represented by brightness (Y) and 
35 color-difference (U,V) may be obtained to obtain the 
three-dimensional shape of the object. In the case 
when the image to be obtained is the black and white 
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iraage, the three-dimensional shape can be obtained 
by the procedure described in the embodiment 5-1 
using the brightness information (Y) as information 
corresponding to the color information so that the 
5 two-dimensional images can be generated. 
(Embodiment 5-2) 

Figs. 101-104 are schematic diagrams for 
explaining the three-dimensional image generation 
method of the embodiment 5-2. Fig. 101 is a diagram 
10 showing relationship between the projection point 
and the corresponding point. Fig. 102 is a flow 
diagram showing an example of steps for determining 
the color information and the existence probability 
of the projection point. Figs. 103 and 104 are 
15 diagrams for explaining a method for obtaining the 
existence probability. 

The basic procedure of the image 
generation method of this embodiment 5-2 is the same 
as the three-dimensional image generation method of 
the embodiment 5-1, and processing from the step 101 
to the step 105 shown in Fig. 90 are performed. The 
different point of the three-dimensional image 
generation method of this embodiment 5-2 compared to 
the three-dimensional image generation method of the 
embodiment 5-1 is that plural images of different 
focusing distances are obtained instead of the 
plural images of different viewpoints in step 101, 
and the three-dimensional shape of the object is 
obtained using the images having different focusing 
30 distances in the step 103. 

For displaying the three-dimensional image 
of the object on the DFD using the three-dimensional 
image generation method of the embodiment 5-2, 
plural images are taken from a viewpoint while 
35 changing the focusing distance, first. At this time, 
the plural images are taken using the polarizing 
binary optical system, a variable focus lens and the 
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like, for example. The image to be obtained may be 
a color image in the same way as the embodiment 5-1 
or may be a black and white image. Next, as 
described in the embodiment 5-1, after setting the 
5 viewpoint of the observer (step 102), processing of 
the step 103 for obtaining the three-dimensional 
shape of the object is performed. 

In the processing of the step 103, as 
described in the embodiment 5-1, projection planes 
10 Lj ( j=l, 2,...,M) and the reference viewpoint R are set 
first (step 10301, step 10302). Then, next, the 
projection point series and the corresponding points 
are set, and the array (area) for storing the 
information of the projection plane is kept (step 

15 10303, step 10304) . 

In the three-dimensional image display 
method of this embodiment 5-2 in which the three- 
dimensional image of the object is displayed using 
plural images having different focusing distances, 
20 when setting the projection planes in step 10301, 
the planes are set such that the distances from the 
viewpoint c of the camera are the same as the 
focusing distances fi <i-l, 2, „., N) of the images taken 
by the camera as shown in Fig, 101, for example. 
25 Then, in the step 10303, the corresponding points Gi 
corresponding to the projection point Tj are 
determined to be points, on images Imgi, overlapping 
with the projection point T 3 when viewing the 
projection point T 3 from the viewpoint C of each 
30 camera. Methods for setting the projection point 
series, and for associating the coordinates of the 
projection point Tj with the digital image 
coordinates of the corresponding point Gi are the 
same as the methods described in the embodiment 5-1. 
35 Thus, detailed descriptions are not provided here. 

In addition, the processing for keeping 
the area for storing the information of the 
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projection planes in the step 10304 is also the same 
as that described in the embodiment 5-1- Thus, 
detailed descriptions are not provided here. 

Next, the color information and the 
5 existence probability information of each projection 
point Tj are determined using the obtained plural 
images (step 10305) . Also in the three-dimensional 
image display method of this embodiment 5-2, in the 
step 10305, double loop processing is performed in 
10 which processing for determining the color 

information and the existence probability of each 
projection point Tj on the projection point series 
is repeated for every projection point series. Thus, 
first, the projection point series is initialized 
15 (step 10305a). Then, next, the projection point Tj 
on the projection point series is initialized to 
for example (step 10305b) . 

Next, the color information of the ■ 
projection point Tj is determined (step 10305c). In 
20 the step 10305c, an average value of color 

information of the corresponding points Gi set in 
the step 10303 is determined to be the color 
information Kj of the projection point T j . 

Next, the focusing degree Qj of the 
25 projection point Tj is obtained based on degree 

(focusing degree) by which focus is achieved for a 
point, of the object, appearing on each 
corresponding point Gi corresponding to the 
projection point Tj (step 10305j). The focusing 
30 degree can be determined according to sharpness or 

blurriness at a point or a small region on the image. 
As a calculation method for the focusing degree, 
there are various methods based on Depth from Focus 
theory or Depth from Defocus theory. The focusing 
35 degree Qj can be obtained by comparing sizes of 

local spatial frequency of each corresponding point 
Gi, for example. 
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The Depth from Focus theory or the Depth 
from Defocus theory is a method for analyzing plural 
images having different focusing distance to measure 
the surface shape of the object. In the method, the 
5 surface of the object is estimated to exist at a 

distance corresponding to a focusing distance of an 
image having the highest local spatial frequency 
among the images taken by changing the focusing 
distance, for example. Thus, the focusing degree Qj 
10 of the projection point T 3 is calculated using an 
evaluation function of the local spatial frequency 
represented by the following equation 115, for 
example . 
[Equation 115] 

15 Q=^tt\t i\f{x,y)-A*+p>y+«i 

D x=jc, y=y t [p=-L c q=~ L r ■> 

wherein f indicates a gray scale of a pixel, D 
indicates a constant for normalization that is a 
number of all pixels, (-Lc, -Lr)-(Lc, Lr) and (xi, 
yi)-(xf, yf) indicate small regions for performing 
20 distribution evaluation and smoothing respectively. 

The equation 115 is one example for a 
method for obtaining the focusing degree Q jf and the 
focusing degree Q 3 can be obtained by using 
equations other than the equation 115. 
25 After obtaining the focusing degree Qj by 

the step 10305h, the projection point T 3 is updated, 
and it is checked whether processing of the steps 
10305c and 10305h are performed for every projection 
point on the projection point series that is a 
30 subject for processing (step 10305e) . When there is 
any projection point for which the processing of the 
steps 10305c and 10305h has not been performed, the 
step returns to the step 10305c so that the color 
information K 3 and the focusing degree Q 3 are 
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obtained. 

When the color information and the 
focusing degree Q 3 are obtained for every projection 
point on the projection point series that is a 
5 subject of processing, color information Kj and the 
focusing degree Qj are provided for each projection 
point Tj on the projection point series as shown in 
Fig. 103. The focusing degree Qj of the projection 
point Tj is a degree corresponding to the 
10 correlation degree used for determining the 

existence probability 3 in the embodiment 5-1. 
There may be a case where there is no projection 
point at which the focusing degree has the 
distinctive small value when the focusing degrees Q 3 
15 of each projection point Tj on the projection point 
series are compared depending on the shape of the 
object, the texture of the surface, photographing 
conditions and the like. In this case, even if the 
surface of the object is estimated to exist at a 
20 projection point, the reliability is low and there 
is a case the estimation is erroneous. When the 
estimation is erroneous, the effect appears as a 
large noise on the generated image. 

Thus, according to the three-dimensional 
25 image display method of this invention, next, the 
probability (existence probability) pj that the 
surface of the object exists on each projection 
point T 3 on the projection point series is 
determined (step 10305f ) - In this case, it is 
30 necessary that the existence probability Pj 

satisfies the equations 106 and 107. Thus, in this 
embodiment 5-2, the existence probability Pj of the 
projection point T k is determined using the equation 
116, for example. 
35 [Equation 116] 
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Basically, it is only necessary that the 
existence probability p, satisfies the conditions of 
the equations 106 and 107. Therefore, the existence 
5 probability pj can be determined using equations 
other than the equation 116. 

After calculating the existence 
probability pj in step 10305f as shown in Fig. 104, 
next, the color information Kj and the existence 
10 probability p 3 of each projection point Tj are stored 
in the area kept in the step 10304. 

After storing the color information Kj and 
the existence probability pj of each projection 
point Tj, the projection point series is updated and 
15 it is checked whether processing from the step 

10305c to the step I0305f has been performed for 
every projection point series determined in the step 
10303 (step 10305g) . When there is any projection 
point series for which processing from the step 
20 10305c to the step I0305f has not been performed, 
the step returns to the step 10305b so that the 
processing from the step 10305c to the step 10305f 
is repeated. 

Accordingly, after the processing from the 
25 step 10305c to the step 10305f are performed for 

every projection point series determined in the step 
10303, the processing of the step 10305 completes, 
so that the three-dimensional shape of the object 
can be obtained. Then, after obtaining the three- 
30 dimensional shape of the object by the processing of 
the step 103, color information and the brightness 
distribution coefficient y of the display point A on 
the two-dimensional image generation plane LD are 
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determined based on the three-dimensional shape of 
the object, so as to generate the two-dimensional 
images to be displayed on the plural overlapping 
image display planes such as the DFD according to 
5 the same procedure as the embodiment 5-1 (step 104) . 
Then, by displaying the generated images on actual 
image display planes (step 105), the three- 
dimensional image of the object can be presented. 

Also in the three-dimensional image 
10 display method of this embodiment 5-2, like the 
three-dimensional image display method of the 
embodiment 5-1, when there is no projection point 
having the focusing degree Qj of a distinctive value 
in the projection points T 3 on the projection point 
15 series, so that reliability for estimation of the 
distance of the surface of the object is low, the 
surface of the object is represented vaguely on 
plural projection planes on the projection point 
series. Then, the brightness distribution 
20 coefficient y of points on the two-dimensional image 
generation plane LD is determined from the existence 
probability P of each projection point T 5 . 
Accordingly, when the two-dimensional images 
generated on the two-dimensional image generation 
25 planes are displayed on the actual image display 

planes so as to present the three-dimensional image 
of the object, the surface of the object is 
displayed vaguely on the projection point series in 
which reliability for distance estimation is low and 
30 the existence probability P is dispersed to plural 

projection points. Therefore, a noise on the three- 
dimensional image displayed on the DFD becomes 
inconspicuous so that an image that looks natural 
for the observer can be displayed. 
35 As described above, according to the 

three-dimensional image generation method of this 
embodiment 5-2, the three-dimensional image that 
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looks natural for the observer can be displayed 
without obtaining the accurate three-dimensional 
shape of the object to be displayed. 

In addition, also in the case of the image 
5 generation method of this embodiment 5-2 , the image 
to be obtained may be either of the color image or a 
black and white image. When using the black and 
white image, processing described in this embodiment 
5-2 is performed using the brightness information 
10 ( Y) as the information corresponding to the color 
information . 

(Embodiment 5-3) 

Also in this embodiment, a three- 
dimensional image generation apparatus having a 
15 configuration similar to that shown in Fig- 85 in the 
fourth embodiment can be provided. In addition, an 
image display system having a configuration similar 
to that shown in Figs. 87-89 in the fourth embodiment 
can be provided. In the configurations of this 
20 embodiment, processing performed by the apparatuses 
corresponds to the embodiments 5-1 and 5-2. 

As mentioned above, although the present 
invention is described concretely based on 
embodiments, the present invention is not limited to 
25 the specifically disclosed embodiments, and 

variations and modifications may be made without 
departing from the scope of the invention. 

For example, although a method for 
displaying the three-dimensional image of the object 
30 based on images of different viewpoints is described 
in the embodiments 5^1, and a method for displaying 
the three-dimensional image of the object based on 
images having different focusing distances is 
described in the embodiment 5-2, the three- 
35 dimensional image of the object can be displayed by 
combining these methods. In this case, the 
correlation degree is obtained from corresponding 
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points of images of different viewpoints with 
respect to a projection point Tj, the local space 
frequency is obtained from corresponding points of 
images taken by changing focal positions from a 
5 viewpoint and the existence probability pj is 
obtained by combining them. By doing so, the 
reliability of the existence probability (Sj 
increases so that an image that looks natural for 
the observer can be displayed, 

10 (Effects of the fifth embodiment) 

Also in the three-dimensional image 
generation method of the fifth embodiment, when 
viewing a direction from the reference viewpoint, 
even when the reliability for estimating a distance 

15 (projection point) at which the object surface 

exists is low, the surface of the object exists with 
a probability on the projection point at which the 
surface of the object actually exists. As a result, 
by displaying each point on the image display plane 

20 with brightness according to the existence 

probability, discontinuous noise that may occur in 
the conventional method when erroneous distance 
estimation is performed can be made inconspicuous. 
In addition, even by an apparatus having low 

25 processing performance such a generally used 

personal computer, each two-dimensional image can be 
generated at high speed. 

The present invention is not limited to 
the specifically disclosed embodiments, and 

30 variations and modifications may be made without 
departing from the scope of the invention. 
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